Task flow identification based on user intent

ABSTRACT

The intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/394,162, entitled “Task Flow Identification Based on User Intent,”filed Dec. 29, 2016 (now granted as U.S. Pat. No. 10,706,841), which isa continuation of U.S. application Ser. No. 13/492,809, entitled“Intelligent Automated Assistant,” filed Jun. 9, 2012 (now granted asU.S. Pat. No. 9,548,050), which is a continuation of U.S. applicationSer. No. 12/987,982, entitled “Intelligent Automated Assistant,” filedJan. 10, 2011 (now granted as U.S. Pat. No. 9,318,108), which claims thebenefit of U.S. Provisional Patent Application Ser. No. 61/295,774,entitled “Intelligent Automated Assistant”, filed Jan. 18, 2010, each ofwhich are hereby incorporated by reference in their entirety for allpurposes.

This application is further related to U.S. patent application Ser. No.11/518,292 for “Method and Apparatus for Building an IntelligentAutomated Assistant”, filed Sep. 8, 2006, which is incorporated hereinby reference.

This application is further related to U.S. Provisional PatentApplication Ser. No. 61/186,414 for “System and Method for SemanticAuto-Completion”, filed Jun. 12, 2009, which is incorporated herein byreference.

FIELD OF THE INVENTION

The present invention relates to intelligent systems, and morespecifically for classes of applications for intelligent automatedassistants.

BACKGROUND OF THE INVENTION

Today's electronic devices are able to access a large, growing, anddiverse quantity of functions, services, and information, both via theInternet and from other sources. Functionality for such devices isincreasing rapidly, as many consumer devices, smartphones, tabletcomputers, and the like, are able to run software applications toperform various tasks and provide different types of information. Often,each application, function, website, or feature has its own userinterface and its own operational paradigms, many of which can beburdensome to learn or overwhelming for users. In addition, many usersmay have difficulty even discovering what functionality and/orinformation is available on their electronic devices or on variouswebsites; thus, such users may become frustrated or overwhelmed, or maysimply be unable to use the resources available to them in an effectivemanner.

In particular, novice users, or individuals who are impaired or disabledin some manner, and/or are elderly, busy, distracted, and/or operating avehicle may have difficulty interfacing with their electronic deviceseffectively, and/or engaging online services effectively. Such users areparticularly likely to have difficulty with the large number of diverseand inconsistent functions, applications, and websites that may beavailable for their use.

Accordingly, existing systems are often difficult to use and tonavigate, and often present users with inconsistent and overwhelminginterfaces that often prevent the users from making effective use of thetechnology.

SUMMARY

According to various embodiments of the present invention, anintelligent automated assistant is implemented on an electronic device,to facilitate user interaction with a device, and to help the user moreeffectively engage with local and/or remote services. In variousembodiments, the intelligent automated assistant engages with the userin an integrated, conversational manner using natural language dialog,and invokes external services when appropriate to obtain information orperform various actions.

According to various embodiments of the present invention, theintelligent automated assistant integrates a variety of capabilitiesprovided by different software components (e.g., for supporting naturallanguage recognition and dialog, multimodal input, personal informationmanagement, task flow management, orchestrating distributed services,and the like). Furthermore, to offer intelligent interfaces and usefulfunctionality to users, the intelligent automated assistant of thepresent invention may, in at least some embodiments, coordinate thesecomponents and services. The conversation interface, and the ability toobtain information and perform follow-on task, are implemented, in atleast some embodiments, by coordinating various components such aslanguage components, dialog components, task management components,information management components and/or a plurality of externalservices.

According to various embodiments of the present invention, intelligentautomated assistant systems may be configured, designed, and/or operableto provide various different types of operations, functionalities,and/or features, and/or to combine a plurality of features, operations,and applications of an electronic device on which it is installed. Insome embodiments, the intelligent automated assistant systems of thepresent invention can perform any or all of: actively eliciting inputfrom a user, interpreting user intent, disambiguating among competinginterpretations, requesting and receiving clarifying information asneeded, and performing (or initiating) actions based on the discernedintent. Actions can be performed, for example, by activating and/orinterfacing with any applications or services that may be available onan electronic device, as well as services that are available over anelectronic network such as the Internet. In various embodiments, suchactivation of external services can be performed via APIs or by anyother suitable mechanism. In this manner, the intelligent automatedassistant systems of various embodiments of the present invention canunify, simplify, and improve the user's experience with respect to manydifferent applications and functions of an electronic device, and withrespect to services that may be available over the Internet. The usercan thereby be relieved of the burden of learning what functionality maybe available on the device and on web-connected services, how tointerface with such services to get what he or she wants, and how tointerpret the output received from such services; rather, the assistantof the present invention can act as a go-between between the user andsuch diverse services.

In addition, in various embodiments, the assistant of the presentinvention provides a conversational interface that the user may findmore intuitive and less burdensome than conventional graphical userinterfaces. The user can engage in a form of conversational dialog withthe assistant using any of a number of available input and outputmechanisms, such as for example speech, graphical user interfaces(buttons and links), text entry, and the like. The system can beimplemented using any of a number of different platforms, such as deviceAPIs, the web, email, and the like, or any combination thereof. Requestsfor additional input can be presented to the user in the context of sucha conversation. Short and long term memory can be engaged so that userinput can be interpreted in proper context given previous events andcommunications within a given session, as well as historical and profileinformation about the user.

In addition, in various embodiments, context information derived fromuser interaction with a feature, operation, or application on a devicecan be used to streamline the operation of other features, operations,or applications on the device or on other devices. For example, theintelligent automated assistant can use the context of a phone call(such as the person called) to streamline the initiation of a textmessage (for example to determine that the text message should be sentto the same person, without the user having to explicitly specify therecipient of the text message). The intelligent automated assistant ofthe present invention can thereby interpret instructions such as “sendhim a text message”, wherein the “him” is interpreted according tocontext information derived from a current phone call, and/or from anyfeature, operation, or application on the device. In variousembodiments, the intelligent automated assistant takes into accountvarious types of available context data to determine which address bookcontact to use, which contact data to use, which telephone number to usefor the contact, and the like, so that the user need not re-specify suchinformation manually.

In various embodiments, the assistant can also take into accountexternal events and respond accordingly, for example, to initiateaction, initiate communication with the user, provide alerts, and/ormodify previously initiated action in view of the external events. Ifinput is required from the user, a conversational interface can again beused.

In one embodiment, the system is based on sets of interrelated domainsand tasks, and employs additional functionally powered by externalservices with which the system can interact. In various embodiments,these external services include web-enabled services, as well asfunctionality related to the hardware device itself. For example, in anembodiment where the intelligent automated assistant is implemented on asmartphone, personal digital assistant, tablet computer, or otherdevice, the assistant can control many operations and functions of thedevice, such as to dial a telephone number, send a text message, setreminders, add events to a calendar, and the like.

In various embodiments, the system of the present invention can beimplemented to provide assistance in any of a number of differentdomains. Examples include:

-   -   Local Services (including location- and time-specific services        such as restaurants, movies, automated teller machines (ATMs),        events, and places to meet);    -   Personal and Social Memory Services (including action items,        notes, calendar events, shared links, and the like);    -   E-commerce (including online purchases of items such as books,        DVDs, music, and the like);    -   Travel Services (including flights, hotels, attractions, and the        like).

One skilled in the art will recognize that the above list of domains ismerely exemplary. In addition, the system of the present invention canbe implemented in any combination of domains.

In various embodiments, the intelligent automated assistant systemsdisclosed herein may be configured or designed to include functionalityfor automating the application of data and services available over theInternet to discover, find, choose among, purchase, reserve, or orderproducts and services. In addition to automating the process of usingthese data and services, at least one intelligent automated assistantsystem embodiment disclosed herein may also enable the combined use ofseveral sources of data and services at once. For ex-ample, it maycombine information about products from several review sites, checkprices and availability from multiple distributors, and check theirlocations and time constraints, and help a user find a personalizedsolution to their problem. Additionally, at least one intelligentautomated assistant system embodiment disclosed herein may be configuredor designed to include functionality for automating the use of data andservices available over the Internet to discover, investigate, selectamong, reserve, and otherwise learn about things to do (including butnot limited to movies, events, performances, exhibits, shows andat-tractions); places to go (including but not limited to traveldestinations, hotels and other places to stay, landmarks and other sitesof interest, etc.); places to eat or drink (such as restaurants andbars), times and places to meet others, and any other source ofentertainment or social interaction which may be found on the Internet.Additionally, at least one intelligent automated assistant systemembodiment disclosed herein may be configured or designed to includefunctionality for enabling the operation of applications and servicesvia natural language dialog that may be otherwise provided by dedicatedapplications with graphical user interfaces including search (includinglocation-based search); navigation (maps and directions); databaselookup (such as finding businesses or people by name or otherproperties); getting weather conditions and forecasts, checking theprice of market items or status of financial transactions; monitoringtraffic or the status of flights; accessing and updating calendars andschedules; managing reminders, alerts, tasks and projects; communicatingover email or other messaging platforms; and operating devices locallyor remotely (e.g., dialing telephones, controlling light andtemperature, controlling home security devices, playing music or video,etc.). Further, at least one intelligent automated assistant systemembodiment disclosed herein may be configured or designed to includefunctionality for identifying, generating, and/or providing personalizedrecommendations for activities, products, services, source ofentertainment, time management, or any other kind of recommendationservice that benefits from an interactive dialog in natural language andautomated access to data and services.

In various embodiments, the intelligent automated assistant of thepresent invention can control many features and operations of anelectronic device. For example, the intelligent automated assistant cancall services that interface with functionality and applications on adevice via APIs or by other means, to perform functions and operationsthat might otherwise be initiated using a conventional user interface onthe device. Such functions and operations may include, for example,setting an alarm, making a telephone call, sending a text message oremail message, adding a calendar event, and the like. Such functions andoperations may be performed as add-on functions in the context of aconversational dialog between a user and the assistant. Such functionsand operations can be specified by the user in the context of such adialog, or they may be automatically performed based on the context ofthe dialog. One skilled in the art will recognize that the assistant canthereby be used as a control mechanism for initiating and controllingvarious operations on the electronic device, which may be used as analternative to conventional mechanisms such as buttons or graphical userinterfaces.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate several embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention according to the embodiments. One skilled inthe art will recognize that the particular embodiments illustrated inthe drawings are merely exemplary, and are not intended to limit thescope of the present invention.

FIG. 1 is a block diagram depicting an example of one embodiment of anintelligent automated assistant system.

FIG. 2 illustrates an example of an interaction between a user and anintelligent automated assistant according to at least one embodiment.

FIG. 3 is a block diagram depicting a computing device suitable forimplementing at least a portion of an intelligent automated assistantaccording to at least one embodiment.

FIG. 4 is a block diagram depicting an architecture for implementing atleast a portion of an intelligent automated assistant on a standalonecomputing system, according to at least one embodiment.

FIG. 5 is a block diagram depicting an architecture for implementing atleast a portion of an intelligent automated assistant on a distributedcomputing network, according to at least one embodiment.

FIG. 6 is a block diagram depicting a system architecture illustratingseveral different types of clients and modes of operation.

FIG. 7 is a block diagram depicting a client and a server, whichcommunicate with each other to implement the present invention accordingto one embodiment.

FIG. 8 is a block diagram depicting a fragment of an active ontologyac-cording to one embodiment.

FIG. 9 is a block diagram depicting an example of an alternativeembodiment of an intelligent automated assistant system.

FIG. 10 is a flow diagram depicting a method of operation for activeinput elicitation component(s) according to one embodiment.

FIG. 11 is a flow diagram depicting a method for active typed-inputelicitation according to one embodiment.

FIGS. 12 to 21 are screen shots illustrating some portions of some ofthe procedures for active typed-input elicitation according to oneembodiment.

FIG. 22 is a flow diagram depicting a method for active inputelicitation for voice or speech input according to one embodiment.

FIG. 23 is a flow diagram depicting a method for active inputelicitation for GUI-based input according to one embodiment.

FIG. 24 is a flow diagram depicting a method for active inputelicitation at the level of a dialog flow according to one embodiment.

FIG. 25 is a flow diagram depicting a method for active monitoring forrelevant events according to one embodiment.

FIG. 26 is a flow diagram depicting a method for multimodal active inputelicitation according to one embodiment.

FIG. 27 is a set of screen shots illustrating an example of varioustypes of functions, operations, actions, and/or other features which maybe provided by domain models component(s) and services orchestrationaccording to one embodiment.

FIG. 28 is a flow diagram depicting an example of a method for naturallanguage processing according to one embodiment.

FIG. 29 is a screen shot illustrating natural language processingaccording to one embodiment.

FIGS. 30 and 31 are screen shots illustrating an example of varioustypes of functions, operations, actions, and/or other features which maybe provided by dialog flow processor component(s) according to oneembodiment.

FIG. 32 is a flow diagram depicting a method of operation for dialogflow processor component(s) according to one embodiment.

FIG. 33 is a flow diagram depicting an automatic call and responseprocedure, according to one embodiment.

FIG. 34 is a flow diagram depicting an example of task flow for aconstrained selection task according to one embodiment.

FIGS. 35 and 36 are screen shots illustrating an example of theoperation of constrained selection task according to one embodiment.

FIG. 37 is a flow diagram depicting an example of a procedure forexecuting a service orchestration procedure according to one embodiment.

FIG. 38 is a flow diagram depicting an example of a service invocationprocedure according to one embodiment.

FIG. 39 is a flow diagram depicting an example of a multiphase outputprocedure according to one embodiment.

FIGS. 40 and 41 are screen shots depicting examples of output processingaccording to one embodiment.

FIG. 42 is a flow diagram depicting an example of multimodal outputprocessing according to one embodiment.

FIGS. 43A and 43B are screen shots depicting an example of the use ofshort term personal memory component(s) to maintain dialog context whilechanging location, according to one embodiment.

FIGS. 44A through 44C are screen shots depicting an example of the useof long term personal memory component(s), according to one embodiment.

FIG. 45 depicts an example of an abstract model for a constrainedselection task.

FIG. 46 depicts an example of a dialog flow model to help guide the userthrough a search process.

FIG. 47 is a flow diagram depicting a method of constrained selectionaccording to one embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Various techniques will now be described in detail with reference to afew example embodiments thereof as illustrated in the accompanyingdrawings. In the following description, numerous specific details areset forth in order to provide a thorough understanding of one or moreaspects and/or features described or reference herein. It will beapparent, however, to one skilled in the art, that one or more aspectsand/or features described or reference herein may be practiced withoutsome or all of these specific details. In other instances, well knownprocess steps and/or structures have not been described in detail inorder to not obscure some of the aspects and/or features described orreference herein.

One or more different inventions may be described in the presentapplication. Further, for one or more of the invention(s) describedherein, numerous embodiments may be described in this patentapplication, and are presented for illustrative purposes only. Thedescribed embodiments are not intended to be limiting in any sense. Oneor more of the invention(s) may be widely applicable to numerousembodiments, as is readily apparent from the disclosure. Theseembodiments are described in sufficient detail to enable those skilledin the art to practice one or more of the invention(s), and it is to beunderstood that other embodiments may be utilized and that structural,logical, software, electrical and other changes may be made withoutdeparting from the scope of the one or more of the invention(s).Accordingly, those skilled in the art will recognize that the one ormore of the invention(s) may be practiced with various modifications andalterations. Particular features of one or more of the invention(s) maybe described with reference to one or more particular embodiments orfigures that form a part of the present disclosure, and in which areshown, by way of illustration, specific embodiments of one or more ofthe invention(s). It should be understood, however, that such featuresare not limited to usage in the one or more particular embodiments orfigures with reference to which they are described. The presentdisclosure is neither a literal description of all embodiments of one ormore of the invention(s) nor a listing of features of one or more of theinvention(s) that must be present in all embodiments.

Headings of sections provided in this patent application and the titleof this patent application are for convenience only, and are not to betaken as limiting the disclosure in any way.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or moreintermediaries.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Tothe contrary, a variety of optional components are described toillustrate the wide variety of possible embodiments of one or more ofthe invention(s).

Further, although process steps, method steps, algorithms or the likemay be described in a sequential order, such processes, methods andalgorithms may be configured to work in alternate orders. In otherwords, any sequence or order of steps that may be described in thispatent application does not, in and of itself, indicate a requirementthat the steps be performed in that order. The steps of describedprocesses may be performed in any order practical. Further, some stepsmay be performed simultaneously despite being described or implied asoccurring non-simultaneously (e.g., because one step is described afterthe other step). Moreover, the illustration of a process by itsdepiction in a drawing does not imply that the illustrated process isexclusive of other variations and modifications thereto, does not implythat the illustrated process or any of its steps are necessary to one ormore of the invention(s), and does not imply that the illustratedprocess is preferred.

When a single device or article is described, it will be readilyapparent that more than one device/article (whether or not theycooperate) may be used in place of a single device/article. Similarly,where more than one device or article is described (whether or not theycooperate), it will be readily apparent that a single device/article maybe used in place of the more than one device or article.

The functionality and/or the features of a device may be alternativelyembodied by one or more other devices that are not explicitly describedas having such functionality/features. Thus, other embodiments of one ormore of the invention(s) need not include the device itself.

Techniques and mechanisms described or reference herein will sometimesbe described in singular form for clarity. However, it should be notedthat particular embodiments include multiple iterations of a techniqueor multiple instantiations of a mechanism unless noted otherwise.

Although described within the context of intelligent automated assistanttechnology, it may be understood that the various aspects and techniquesdescribed herein (such as those associated with active ontologies, forexample) may also be deployed and/or applied in other fields oftechnology involving human and/or computerized interaction withsoftware.

Other aspects relating to intelligent automated assistant technology(e.g., which may be utilized by, provided by, and/or implemented at oneor more intelligent automated assistant system embodiments describedherein) are disclosed in one or more of the following references:

-   -   U.S. Provisional Patent Application Ser. No. 61/295,774 for        “Intelligent Automated Assistant”, filed Jan. 18, 2010, the        disclosure of which is incorporated herein by reference;    -   U.S. patent application Ser. No. 11/518,292 for “Method And        Apparatus for Building an Intelligent Automated Assistant”,        filed Sep. 8, 2006, the disclosure of which is incorporated        herein by reference; and    -   U.S. Provisional Patent Application Ser. No. 61/186,414 for        “System and Method for Semantic Auto-Completion”, filed Jun. 12,        2009, the disclosure of which is incorporated herein by        reference.

Hardware Architecture

Generally, the intelligent automated assistant techniques disclosedherein may be implemented on hardware or a combination of software andhardware. For example, they may be implemented in an operating systemkernel, in a separate user process, in a library package bound intonetwork applications, on a specially constructed machine, or on anetwork interface card. In a specific embodiment, the techniquesdisclosed herein may be implemented in software such as an operatingsystem or in an application running on an operating system.

Software/hardware hybrid implementation(s) of at least some of theintelligent automated assistant embodiment(s) disclosed herein may beimplemented on a programmable machine selectively activated orreconfigured by a computer program stored in memory. Such networkdevices may have multiple network interfaces which may be configured ordesigned to utilize different types of network communication protocols.A general architecture for some of these machines may appear from thedescriptions disclosed herein. According to specific embodiments, atleast some of the features and/or functionalities of the variousintelligent automated assistant embodiments disclosed herein may beimplemented on one or more general-purpose network host machines such asan end-user computer system, computer, network server or server system,mobile computing device (e.g., personal digital assistant, mobile phone,smartphone, laptop, tablet computer, or the like), consumer electronicdevice, music player, or any other suitable electronic device, router,switch, or the like, or any combination thereof. In at least someembodiments, at least some of the features and/or functionalities of thevarious intelligent automated assistant embodiments disclosed herein maybe implemented in one or more virtualized computing environments (e.g.,network computing clouds, or the like).

Referring now to FIG. 3, there is shown a block diagram depicting acomputing device 60 suitable for implementing at least a portion of theintelligent automated assistant features and/or functionalitiesdisclosed herein. Computing device 60 may be, for example, an end-usercomputer system, network server or server system, mobile computingdevice (e.g., personal digital assistant, mobile phone, smartphone,laptop, tablet computer, or the like), consumer electronic device, musicplayer, or any other suitable electronic device, or any combination orportion thereof. Computing device 60 may be adapted to communicate withother computing devices, such as clients and/or servers, over acommunications network such as the Internet, using known protocols forsuch communication, whether wireless or wired.

In one embodiment, computing device 60 includes central processing unit(CPU) 62, interfaces 68, and a bus 67 (such as a peripheral componentinter-connect (PCI) bus). When acting under the control of appropriatesoftware or firmware, CPU 62 may be responsible for implementingspecific functions associated with the functions of a specificallyconfigured computing device or machine. For example, in at least oneembodiment, a user's personal digital assistant (PDA) may be configuredor designed to function as an intelligent automated assistant systemutilizing CPU 62, memory 61, 65, and interface(s) 68. In at least oneembodiment, the CPU 62 may be caused to perform one or more of thedifferent types of intelligent automated assistant functions and/oroperations under the control of software modules/components, which forexample, may include an operating system and any appropriateapplications software, drivers, and the like.

CPU 62 may include one or more processor(s) 63 such as, for example, aprocessor from the Motorola or Intel family of microprocessors or theMIPS family of microprocessors. In some embodiments, processor(s) 63 mayinclude specially designed hardware (e.g., application-specificintegrated circuits (ASICs), electrically erasable programmableread-only memories (EEPROMs), field-programmable gate arrays (FPGAs),and the like) for controlling the operations of computing device 60. Ina specific embodiment, a memory 61 (such as non-volatile random accessmemory (RAM) and/or read-only memory (ROM)) also forms part of CPU 62.However, there are many different ways in which memory may be coupled tothe system. Memory block 61 may be used for a variety of purposes suchas, for example, caching and/or storing data, programming instructions,and the like.

As used herein, the term “processor” is not limited merely to thoseintegrated circuits referred to in the art as a processor, but broadlyrefers to a microcontroller, a microcomputer, a programmable logiccontroller, an application-specific integrated circuit, and any otherprogrammable circuit.

In one embodiment, interfaces 68 are provided as interface cards(sometimes referred to as “line cards”). Generally, they control thesending and receiving of data packets over a computing network andsometimes support other peripherals used with computing device 60. Amongthe interfaces that may be provided are Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces, andthe like. In addition, various types of interfaces may be provided suchas, for example, universal serial bus (USB), Serial, Ethernet, Firewire,PCI, parallel, radio frequency (RF), Bluetooth™, near-fieldcommunications (e.g., using near-field magnetics), 802.11 (WiFi), framerelay, TCP/IP, ISDN, fast Ethernet interfaces, Gigabit Ethernetinterfaces, asynchronous transfer mode (ATM) interfaces, high-speedserial interface (HSSI) interfaces, Point of Sale (POS) interfaces,fiber data distributed interfaces (FDDIs), and the like. Generally, suchinterfaces 68 may include ports appropriate for communication with theappropriate media. In some cases, they may also include an independentprocessor and, in some instances, volatile and/or nonvolatile memory(e.g., RAM).

Although the system shown in FIG. 3 illustrates one specificarchitecture for a computing device 60 for implementing the techniquesof the invention described herein, it is by no means the only devicearchitecture on which at least a portion of the features and techniquesdescribed herein may be implemented. For example, architectures havingone or any number of processors 63 can be used, and such processors 63can be present in a single device or distributed among any number ofdevices. In one embodiment, a single processor 63 handles communicationsas well as routing computations. In various embodiments, different typesof intelligent automated assistant features and/or functionalities maybe implemented in an intelligent automated assistant system whichincludes a client device (such as a personal digital assistant orsmartphone running client software) and server system(s) (such as aserver system described in more detail below).

Regardless of network device configuration, the system of the presentinvention may employ one or more memories or memory modules (such as,for example, memory block 65) configured to store data, programinstructions for the general-purpose network operations and/or otherinformation relating to the functionality of the intelligent automatedassistant techniques described herein. The program instructions maycontrol the operation of an operating system and/or one or moreapplications, for example. The memory or memories may also be configuredto store data structures, keyword taxonomy information, advertisementinformation, user click and impression information, and/or otherspecific non-program information described herein.

Because such information and program instructions may be employed toimplement the systems/methods described herein, at least some networkdevice embodiments may include nontransitory machine-readable storagemedia, which, for example, may be configured or designed to storeprogram instructions, state information, and the like for performingvarious operations described herein. Examples of such nontransitorymachine-readable storage media include, but are not limited to, magneticmedia such as hard disks, floppy disks, and magnetic tape; optical mediasuch as CD-ROM disks; magneto-optical media such as floptical disks, andhardware devices that are specially configured to store and performprogram instructions, such as read-only memory devices (ROM), flashmemory, memristor memory, random access memory (RAM), and the like.Examples of program instructions include both machine code, such asproduced by a compiler, and files containing higher level code that maybe executed by the computer using an interpreter.

In one embodiment, the system of the present invention is implemented ona standalone computing system. Referring now to FIG. 4, there is shown ablock diagram depicting an architecture for implementing at least aportion of an intelligent automated assistant on a standalone computingsystem, according to at least one embodiment. Computing device 60includes processor(s) 63 which run software for implementing intelligentautomated assistant 1002. Input device 1206 can be of any type suitablefor receiving user input, including for example a keyboard, touchscreen, microphone (for example, for voice input), mouse, touchpad,trackball, five-way switch, joystick, and/or any combination thereof.Output device 1207 can be a screen, speaker, printer, and/or anycombination thereof. Memory 1210 can be random-access memory having astructure and architecture as are known in the art, for use byprocessor(s) 63 in the course of running software. Storage device 1208can be any magnetic, optical, and/or electrical storage device forstorage of data in digital form; examples include flash memory, magnetichard drive, CD-ROM, and/or the like.

In another embodiment, the system of the present invention isimplemented on a distributed computing network, such as one having anynumber of clients and/or servers. Referring now to FIG. 5, there isshown a block diagram depicting an architecture for implementing atleast a portion of an intelligent automated assistant on a distributedcomputing network, according to at least one embodiment.

In the arrangement shown in FIG. 5, any number of clients 1304 areprovided; each client 1304 may run software for implementing client-sideportions of the present invention. In addition, any number of servers1340 can be provided for handling requests received from clients 1304.Clients 1304 and servers 1340 can communicate with one another viaelectronic network 1361, such as the Internet. Network 1361 may beimplemented using any known net-work protocols, including for examplewired and/or wireless protocols.

In addition, in one embodiment, servers 1340 can call external services1360 when needed to obtain additional information or refer to store dataconcerning previous interactions with particular users. Communicationswith external services 1360 can take place, for example, via network1361. In various embodiments, external services 1360 include web-enabledservices and/or functionality related to or installed on the hardwaredevice itself. For example, in an embodiment where assistant 1002 isimplemented on a smartphone or other electronic device, assistant 1002can obtain information stored in a calendar application (“app”),contacts, and/or other sources.

In various embodiments, assistant 1002 can control many features andoperations of an electronic device on which it is installed. Forexample, assistant 1002 can call external services 1360 that interfacewith functionality and applications on a device via APIs or by othermeans, to perform functions and operations that might otherwise beinitiated using a conventional user interface on the device. Suchfunctions and operations may include, for example, setting an alarm,making a telephone call, sending a text message or email message, addinga calendar event, and the like. Such functions and operations may beperformed as add-on functions in the context of a conversational dialogbetween a user and assistant 1002. Such functions and operations can bespecified by the user in the context of such a dialog, or they may beautomatically performed based on the context of the dialog. One skilledin the art will recognize that assistant 1002 can thereby be used as acontrol mechanism for initiating and controlling various operations onthe electronic device, which may be used as an alternative toconventional mechanisms such as buttons or graphical user interfaces.

For example, the user may provide input to assistant 1002 such as “Ineed to wake tomorrow at 8 am.” Once assistant 1002 has determined theuser's intent, using the techniques described herein, assistant 1002 cancall external services 1340 to interface with an alarm clock function orapplication on the device. Assistant 1002 sets the alarm on behalf ofthe user. In this manner, the user can use assistant 1002 as areplacement for conventional mechanisms for setting the alarm orperforming other functions on the device. If the user's requests areambiguous or need further clarification, assistant 1002 can use thevarious techniques described herein, including active elicitation,paraphrasing, suggestions, and the like, to obtain the neededinformation so that the correct services 1340 are called and theintended action taken. In one embodiment, assistant 1002 may prompt theuser for confirmation before calling a service 1340 to perform afunction. In one embodiment, a user can selectively disable assistant's1002 ability to call particular services 1340, or can disable all suchservice-calling if desired.

The system of the present invention can be implemented with manydifferent types of clients 1304 and modes of operation. Referring now toFIG. 6, there is shown a block diagram depicting a system architectureillustrating several different types of clients 1304 and modes ofoperation. One skilled in the art will recognize that the various typesof clients 1304 and modes of operation shown in FIG. 6 are merelyexemplary, and that the system of the present invention can beimplemented using clients 1304 and/or modes of operation other thanthose depicted. Additionally, the system can include any or all of suchclients 1304 and/or modes of operation, alone or in any combination.Depicted examples include:

-   -   Computer devices with input/output devices and/or sensors 1402.        A client component may be deployed on any such computer device        1402. At least one embodiment may be implemented using a web        browser 1304A or other software application for enabling        communication with servers 1340 via network 1361. Input and        output channels may of any type, including for example visual        and/or auditory channels. For example, in one embodiment, the        system of the invention can be implemented using voice-based        communication methods, allowing for an embodiment of the        assistant for the blind whose equivalent of a web browser is        driven by speech and uses speech for output.    -   Mobile Devices with I/O and sensors 1406, for which the client        may be implemented as an application on the mobile device 1304B.        This includes, but is not limited to, mobile phones,        smartphones, personal digital assistants, tablet devices,        networked game consoles, and the like.    -   Consumer Appliances with I/O and sensors 1410, for which the        client may be implemented as an embedded application on the        appliance 1304C.    -   Automobiles and other vehicles with dashboard interfaces and        sensors 1414, for which the client may be implemented as an        embedded system application 1304 D. This includes, but is not        limited to, car navigation systems, voice control systems,        in-car entertainment systems, and the like.    -   Networked computing devices such as routers 1418 or any other        device that resides on or interfaces with a network, for which        the client may be implemented as a device-resident application        1304E.    -   Email clients 1424, for which an embodiment of the assistant is        connected via an Email Modality Server 1426. Email Modality        server 1426 acts as a communication bridge, for example taking        input from the user as email messages sent to the assistant and        sending output from the assistant to the user as replies.    -   Instant messaging clients 1428, for which an embodiment of the        assistant is connected via a Messaging Modality Server 1430.        Messaging Modality server 1430 acts as a communication bridge,        taking input from the user as messages sent to the assistant and        sending output from the assistant to the user as messages in        reply.    -   Voice telephones 1432, for which an embodiment of the assistant        is connected via a Voice over Internet Protocol (VoIP) Modality        Server 1430. VoIP Modality server 1430 acts as a communication        bridge, taking input from the user as voice spoken to the        assistant and sending output from the assistant to the user, for        example as synthesized speech, in reply.

For messaging platforms including but not limited to email, instantmessaging, discussion forums, group chat sessions, live help or customersupport sessions and the like, assistant 1002 may act as a participantin the conversations. Assistant 1002 may monitor the conversation andreply to individuals or the group using one or more the techniques andmethods described herein for one-to-one interactions.

In various embodiments, functionality for implementing the techniques ofthe present invention can be distributed among any number of clientand/or server components. For example, various software modules can beimplemented for performing various functions in connection with thepresent invention, and such modules can be variously implemented to runon server and/or client components. Referring now to FIG. 7, there isshown an example of a client 1304 and a server 1340, which communicatewith each other to implement the present invention according to oneembodiment. FIG. 7 depicts one possible arrangement by which softwaremodules can be distributed among client 1304 and server 1340. Oneskilled in the art will recognize that the depicted arrangement ismerely exemplary, and that such modules can be distributed in manydifferent ways. In addition, any number of clients 1304 and/or servers1340 can be provided, and the modules can be distributed among theseclients 1304 and/or servers 1340 in any of a number of different ways.

In the example of FIG. 7, input elicitation functionality and outputprocessing functionality are distributed among client 1304 and server1340, with client part of input elicitation 1094 a and client part ofoutput processing 1092 a located at client 1304, and server part ofinput elicitation 1094 b and server part of output processing 1092 blocated at server 1340. The following components are located at server1340:

-   -   complete vocabulary 1058 b;    -   complete library of language pattern recognizers 1060 b;    -   master version of short term personal memory 1052 b;    -   master version of long term personal memory 1054 b.

In one embodiment, client 1304 maintains subsets and/or portions ofthese components locally, to improve responsiveness and reducedependence on network communications. Such subsets and/or portions canbe maintained and updated according to well known cache managementtechniques. Such subsets and/or portions include, for example:

-   -   subset of vocabulary 1058 a;    -   subset of library of language pattern recognizers 1060 a;    -   cache of short term personal memory 1052 a;    -   cache of long term personal memory 1054 a.

Additional components may be implemented as part of server 1340,including for example:

-   -   language interpreter 1070;    -   dialog flow processor 1080;    -   output processor 1090;    -   domain entity databases 1072;    -   task flow models 1086;    -   services orchestration 1082;    -   service capability models 1088.

Each of these components will be described in more detail below. Server1340 obtains additional information by interfacing with externalservices 1360 when needed.

Conceptual Architecture

Referring now to FIG. 1, there is shown a simplified block diagram of aspecific example embodiment of an intelligent automated assistant 1002.As described in greater detail herein, different embodiments ofintelligent automated assistant systems may be configured, designed,and/or operable to provide various different types of operations,functionalities, and/or features generally relating to intelligentautomated assistant technology. Further, as described in greater detailherein, many of the various operations, functionalities, and/or featuresof the intelligent automated assistant system(s) disclosed herein mayprovide may enable or provide different types of advantages and/orbenefits to different entities interacting with the intelligentautomated assistant system(s). The embodiment shown in FIG. 1 may beimplemented using any of the hardware architectures described above, orusing a different type of hardware architecture.

For example, according to different embodiments, at least someintelligent automated assistant system(s) may be configured, designed,and/or operable to provide various different types of operations,functionalities, and/or features, such as, for example, one or more ofthe following (or combinations thereof):

-   -   automate the application of data and services available over the        Internet to discover, find, choose among, purchase, reserve, or        order products and services. In addition to automating the        process of using these data and services, intelligent automated        assistant 1002 may also enable the combined use of several        sources of data and services at once. For example, it may        combine information about products from several review sites,        check prices and availability from multiple distributors, and        check their locations and time constraints, and help a user find        a personalized solution to their problem.    -   automate the use of data and services available over the        Internet to discover, investigate, select among, reserve, and        otherwise learn about things to do (including but not limited to        movies, events, performances, exhibits, shows and attractions);        places to go (including but not limited to travel destinations,        hotels and other places to stay, landmarks and other sites of        interest, and the like); places to eat or drink (such as        restaurants and bars), times and places to meet others, and any        other source of entertainment or social interaction which may be        found on the Internet.    -   enable the operation of applications and services via natural        language dialog that are otherwise provided by dedicated        applications with graphical user interfaces including search        (including location-based search); navigation (maps and        directions); database lookup (such as finding businesses or        people by name or other properties); getting weather conditions        and forecasts, checking the price of market items or status of        financial transactions; monitoring traffic or the status of        flights; accessing and updating calendars and schedules;        managing reminders, alerts, tasks and projects; communicating        over email or other messaging platforms; and operating devices        locally or remotely (e.g., dialing telephones, controlling light        and temperature, controlling home security devices, playing        music or video, and the like). In one embodiment, assistant 1002        can be used to initiate, operate, and control many functions and        apps available on the device.    -   offer personal recommendations for activities, products,        services, source of entertainment, time management, or any other        kind of recommendation service that benefits from an interactive        dialog in natural language and automated access to data and        services.

According to different embodiments, at least a portion of the varioustypes of functions, operations, actions, and/or other features providedby intelligent automated assistant 1002 may be implemented at one ormore client systems(s), at one or more server systems (s), and/orcombinations thereof.

According to different embodiments, at least a portion of the varioustypes of functions, operations, actions, and/or other features providedby assistant 1002 may implement by at least one embodiment of anautomated call and response procedure, such as that illustrated anddescribed, for example, with respect to FIG. 33.

Additionally, various embodiments of assistant 1 002 described hereinmay include or provide a number of different advantages and/or benefitsover currently existing intelligent automated assistant technology suchas, for example, one or more of the following (or combinations thereof):

-   -   The integration of speech-to-text and natural language        understanding technology that is constrained by a set of        explicit models of domains, tasks, services, and dialogs. Unlike        assistant technology that attempts to implement a        general-purpose artificial intelligence system, the embodiments        described herein may apply the multiple sources of constraints        to reduce the number of solutions to a more tractable size. This        results in fewer ambiguous interpretations of language, fewer        relevant domains or tasks, and fewer ways to operationalize the        intent in services. The focus on specific domains, tasks, and        dialogs also makes it feasible to achieve coverage over domains        and tasks with human-managed vocabulary and mappings from intent        to services parameters.    -   The ability to solve user problems by invoking services on their        behalf over the Internet, using APIs. Unlike search engines        which only return links and content, some embodiments of        automated assistants 1002 described herein may automate research        and problem-solving activities. The ability to invoke multiple        services for a given request also provides broader functionality        to the user than is achieved by visiting a single site, for        instance to produce a product or service or find something to        do.    -   The application of personal information and personal interaction        history in the interpretation and execution of user requests.        Unlike conventional search engines or question answering        services, the embodiments described herein use information from        personal interaction history (e.g., dialog history, previous        selections from results, and the like), personal physical        context (e.g., user's location and time), and personal        information gathered in the context of interaction (e.g., name,        email addresses, physical addresses, phone numbers, account        numbers, preferences, and the like). Using these sources of        information enables, for example,        -   better interpretation of user input (e.g., using personal            history and physical context when interpreting language);        -   more personalized results (e.g., that bias toward            preferences or recent selections);        -   improved efficiency for the user (e.g., by automating steps            involving the signing up to services or filling out forms).    -   The use of dialog history in interpreting the natural language        of user inputs. Because the embodiments may keep personal        history and apply natural language understanding on user inputs,        they may also use dialog context such as current location, time,        domain, task step, and task parameters to interpret the new        inputs. Conventional search engines and command processors        interpret at least one query independent of a dialog history.        The ability to use dialog history may make a more natural        interaction possible, one which resembles normal human        conversation.    -   Active input elicitation, in which assistant 1002 actively        guides and constrains the input from the user, based on the same        models and information used to interpret their input. For        example, assistant 1 002 may apply dialog models to suggest next        steps in a dialog with the user in which they are refining a        request; offer completions to partially typed input based on        domain and context specific possibilities; or use semantic        interpretation to select from among ambiguous interpretations of        speech as text or text as intent.    -   The explicit modeling and dynamic management of services, with        dynamic and robust services orchestration. The architecture of        embodiments described enables assistant 1 002 to interface with        many external services, dynamically determine which services may        provide information for a specific user request, map parameters        of the user request to different service APIs, call multiple        services at once, integrate results from multiple services, fail        over gracefully on failed services, and/or efficiently maintain        the implementation of services as their APIs and capabilities        evolve.    -   The use of active ontologies as a method and apparatus for        building assistants 1002, which simplifies the software        engineering and data maintenance of automated assistant systems.        Active ontologies are an integration of data modeling and        execution environments for assistants. They provide a framework        to tie together the various sources of models and data (domain        concepts, task flows, vocabulary, language pattern recognizers,        dialog context, user personal information, and mappings from        domain and task requests to external services. Active ontologies        and the other architectural innovations described herein make it        practical to build deep functionality within domains, unifying        multiple sources of information and services, and to do this        across a set of domains.

In at least one embodiment, intelligent automated assistant 1 002 may beoperable to utilize and/or generate various different types of dataand/or other types of information when performing specific tasks and/oroperations. This may include, for example, input data/information and/oroutput data/information. For example, in at least one embodiment,intelligent automated assistant 1002 may be operable to access, process,and/or otherwise utilize information from one or more different types ofsources, such as, for example, one or more local and/or remote memories,devices and/or systems. Additionally, in at least one embodiment,intelligent automated assistant 1002 may be operable to generate one ormore different types of output data/information, which, for example, maybe stored in memory of one or more local and/or remote devices and/orsystems.

Examples of different types of input data/information which may beaccessed and/or utilized by intelligent automated assistant 1002 mayinclude, but are not limited to, one or more of the following (orcombinations thereof):

-   -   Voice input: from mobile devices such as mobile telephones and        tablets, computers with microphones, Bluetooth headsets,        automobile voice control systems, over the telephone system,        recordings on answering services, audio voicemail on integrated        messaging services, consumer applications with voice input such        as clock radios, telephone station, home entertainment control        systems, and game consoles.    -   Text input from keyboards on computers or mobile devices,        keypads on remote controls or other consumer electronics        devices, email messages sent to the assistant, instant messages        or similar short messages sent to the assistant, text received        from players in multiuser game environments, and text streamed        in message feeds.    -   Location information coming from sensors or location-based        systems. Examples include Global Positioning System (GPS) and        Assisted GPS (A-GPS) on mobile phones. In one embodiment,        location information is combined with explicit user input. In        one embodiment, the system of the present invention is able to        detect when a user is at home, based on known address        information and current location determination. In this manner,        certain inferences may be made about the type of information the        user might be interested in when at home as opposed to outside        the home, as well as the type of services and actions that        should be invoked on behalf of the user depending on whether or        not he or she is at home.    -   Time information from clocks on client devices. This may        include, for example, time from telephones or other client        devices indicating the local time and time zone. In addition,        time may be used in the context of user requests, such as for        instance, to interpret phrases such as “in an hour” and        “tonight”.    -   Compass, accelerometer, gyroscope, and/or travel velocity data,        as well as other sensor data from mobile or handheld devices or        embedded systems such as automobile control systems. This may        also include device positioning data from remote controls to        appliances and game consoles.    -   Clicking and menu selection and other events from a graphical        user interface (GUI) on any device having a GUI. Further        examples include touches to a touch screen.    -   Events from sensors and other data-driven triggers, such as        alarm clocks, calendar alerts, price change triggers, location        triggers, push notification onto a device from servers, and the        like.

The input to the embodiments described herein also includes the contextof the user interaction history, including dialog and request history.

Examples of different types of output data/information which may begenerated by intelligent automated assistant 1002 may include, but arenot limited to, one or more of the following (or combinations thereof):

-   -   Text output sent directly to an output device and/or to the user        interface of a device    -   Text and graphics sent to a user over email    -   Text and graphics send to a user over a messaging service    -   Speech output, may include one or more of the following (or        combinations thereof):        -   Synthesized speech        -   Sampled speech        -   Recorded messages    -   Graphical layout of information with photos, rich text, videos,        sounds, and hyperlinks. For instance, the content rendered in a        web browser.    -   Actuator output to control physical actions on a device, such as        causing it to turn on or off, make a sound, change color,        vibrate, control a light, or the like.    -   Invoking other applications on a device, such as calling a        mapping application, voice dialing a telephone, sending an email        or instant message, playing media, making entries in calendars,        task managers, and note applications, and other applications.    -   Actuator output to control physical actions to devices attached        or controlled by a device, such as operating a remote camera,        controlling a wheelchair, playing music on remote speakers,        playing videos on remote displays, and the like.

It may be appreciated that the intelligent automated assistant 1002 ofFIG. 1 is but one example from a wide range of intelligent automatedassistant system embodiments which may be implemented. Other embodimentsof the intelligent automated assistant system (not shown) may includeadditional, fewer and/or different components/features than thoseillustrated, for example, in the example intelligent automated assistantsystem embodiment of FIG. 1.

User Interaction

Referring now to FIG. 2, there is shown an example of an interactionbetween a user and at least one embodiment of an intelligent automatedassistant 1002. The example of FIG. 2 assumes that a user is speaking tointelligent automated assistant 1002 using input device 1206, which maybe a speech input mechanism, and the output is graphical layout tooutput device 1207, which may be a scrollable screen. Conversationscreen 1 0 1 A features a conversational user interface showing what theuser said 1018 (“I'd like a romantic place for Italian food near myoffice”) and assistant's 1002 response, which is a summary of itsfindings 101C (“OK, I found these Italian restaurants which reviews sayare romantic close to your work:”) and a set of results 101 D (the firstthree of a list of restaurants are shown). In this example, the userclicks on the first result in the list, and the result automaticallyopens up to reveal more information about the restaurant, shown ininformation screen 101E. Information screen 101E and conversation screen101A may appear on the same output device, such as a touch-screen orother display device; the examples depicted in FIG. 2 are two differentoutput states for the same output device.

In one embodiment, information screen 101E shows information gatheredand combined from a variety of services, including for example, any orall of the following:

-   -   Addresses and geolocations of businesses;    -   Distance from user's current location;    -   Reviews from a plurality of sources;

In one embodiment, information screen 101E also includes some examplesof services that assistant 1002 might offer on behalf of the user,including:

-   -   Dial a telephone to call the business (“call”);    -   Remember this restaurant for future reference (“save”);    -   Send an email to someone with the directions and information        about this restaurant (“share”);    -   Show the location of and directions to this restaurant on a map        (“map it”);    -   Save personal notes about this restaurant (“my notes”).

As shown in the example of FIG. 2, in one embodiment, assistant 1002includes intelligence beyond simple database applications, such as, forexample,

-   -   Processing a statement of intent in a natural language 101B, not        just keywords;    -   Inferring semantic intent from that language input, such as        interpreting “place for Italian food” as “Italian restaurants”;    -   Operationalizing semantic intent into a strategy for using        online services and executing that strategy on behalf of the        user (e.g., operationalizing the desire for a romantic place        into the strategy of checking on-line review sites for reviews        that describe a place as “romantic”).

Intelligent Automated Assistant Components

According to various embodiments, intelligent automated assistant 1 002may include a plurality of different types of components, devices,modules, processes, systems, and the like, which, for example, may beimplemented and/or instantiated via the use of hardware and/orcombinations of hardware and software. For example, as illustrated inthe example embodiment of FIG. 1, assistant 1 002 may include one ormore of the following types of systems, components, devices, processes,and the like (or combinations thereof):

-   -   One or more active ontologies 1050;    -   Active input elicitation component(s) 1094 (may include client        part 1094 a and server part 1094 b);    -   Short term personal memory component(s) 1052 (may include master        version 1052 b and cache 1052 a);    -   Long-term personal memory component(s) 1054 (may include master        version 1052 b and cache 1052 a);    -   Domain models component(s) 1056;    -   Vocabulary component(s) 1058 (may include complete vocabulary        1058 b and subset 1058 a);    -   Language pattern recognizer(s) component(s) 1060 (may include        full library 1060 b and subset 1560 a);    -   Language interpreter component(s) 1070;    -   Domain entity database(s) 1072;    -   Dialog flow processor component(s) 1080;    -   Services orchestration component(s) 1082;    -   Services component(s) 1084;    -   Task flow models component(s) 1086;    -   Dialog flow models component(s) 1087;    -   Service models component(s) 1088;    -   Output processor component(s) 1090.

As described in connection with FIG. 7, in certain client/server-basedembodiments, some or all of these components may be distributed betweenclient 1304 and server 1340.

For purposes of illustration, at least a portion of the different typesof components of a specific example embodiment of intelligent automatedassistant 1002 will now be described in greater detail with reference tothe example intelligent automated assistant 1002 embodiment of FIG. 1.

Active Ontologies 1 050

Active ontologies 1050 serve as a unifying infrastructure thatintegrates models, components, and/or data from other parts ofembodiments of intelligent automated assistants 1002. In the field ofcomputer and information science, ontologies provide structures for dataand knowledge representation such as classes/types, relations,attributes/properties and their instantiation in instances. Ontologiesare used, for example, to build models of data and knowledge. In someembodiments of the intelligent automated system 1002, ontologies arepart of the modeling framework in which to build models such as domainmodels.

Within the context of the present invention, an “active ontology” 1050may also serve as an execution environment, in which distinct processingelements are arranged in an ontology-like manner (e.g., having distinctattributes and relations with other processing elements). Theseprocessing elements carry out at least some of the tasks of intelligentautomated assistant 1002. Any number of active ontologies 1050 can beprovided.

In at least one embodiment, active ontologies 1050 may be operable toperform and/or implement various types of functions, operations,actions, and/or other features such as, for example, one or more of thefollowing (or combinations thereof):

-   -   Act as a modeling and development environment, integrating        models and data from various model and data components,        including but not limited to        -   Domain models 1056        -   Vocabulary 1058        -   Domain entity databases 1072        -   Task flow models 1086        -   Dialog flow models 1087        -   Service capability models 1088    -   Act as a data-modeling environment on which ontology-based        editing tools may operate to develop new models, data        structures, database schemata, and representations.    -   Act as a live execution environment, instantiating values for        elements of domain 1056, task 1086, and/or dialog models 1087,        language pattern recognizers, and/or vocabulary 1058, and        user-specific information such as that found in short term        personal memory 1052, long term personal memory 1054, and/or the        results of service orchestration 1182. For example, some nodes        of an active ontology may correspond to domain concepts such as        restaurant and its property restaurant name. During live        execution, these active ontology nodes may be instantiated with        the identity of a particular restaurant entity and its name, and        how its name corresponds to words in a natural language input        utterance. Thus, in this embodiment, the active ontology is        serving as both a modeling environment specifying the concept        that restaurants are entities with identities that have names,        and for storing dynamic bindings of those modeling nodes with        data from entity databases and parses of natural language.    -   Enable the communication and coordination among components and        processing elements of an intelligent automated assistant, such        as, for example, one or more of the following (or combinations        thereof):        -   Active input elicitation component(s) 1094        -   Language interpreter component(s) 1070        -   Dialog flow processor component(s) 1080        -   Services orchestration component(s) 1082        -   Services component(s) 1084

In one embodiment, at least a portion of the functions, operations,actions, and/or other features of active ontologies 1050 describedherein may be implemented, at least in part, using various methods andapparatuses described in U.S. patent application Ser. No. 11/518,292 for“Method and Apparatus for Building an Intelligent Automated Assistant”,filed Sep. 8, 2006.

In at least one embodiment, a given instance of active ontology 1050 mayaccess and/or utilize information from one or more associated databases.In at least one embodiment, at least a portion of the databaseinformation may be accessed via communication with one or more localand/or remote memory devices. Examples of different types of data whichmay be accessed by active ontologies 1050 may include, but are notlimited to, one or more of the following (or combinations thereof):

-   -   Static data that is available from one or more components of        intelligent automated assistant 1002;    -   Data that is dynamically instantiated per user session, for        example, but not limited to, maintaining the state of the        user-specific inputs and outputs exchanged among components of        intelligent automated assistant 1002, the contents of short term        personal memory, the inferences made from previous states of the        user session, and the like.

In this manner, active ontologies 1050 are used to unify elements ofvarious components in intelligent automated assistant 1002. An activeontology 1050 allows an author, designer, or system builder to integratecomponents so that the elements of one component are identified withelements of other components. The author, designer, or system buildercan thus combine and integrate the components more easily.

Referring now to FIG. 8, there is shown an example of a fragment of anactive ontology 1050 according to one embodiment. This example isintended to help illustrate some of the various types of functions,operations, actions, and/or other features that may be provided byactive ontologies 1050.

Active ontology 1050 in FIG. 8 includes representations of a restaurantand meal event. In this example, a restaurant is a concept 1610 withproperties such as its name 1612, cuisines served 1615, and its location1613, which in turn might be modeled as a structured node withproperties for street address 1614. The concept of a meal event might bemodeled as a node 1616 including a dining party 1617 (which has a size1619) and time period 1618.

-   -   Active ontologies may include and/or make reference to domain        models 1056. For example, FIG. 8 depicts a dining out domain        model 1622 linked to restaurant concept 1610 and meal event        concept 1616. In this instance, active ontology 1050 includes        dining out domain model 1622; specifically, at least two nodes        of active ontology 1050, namely restaurant 1610 and meal event        1616, are also included in and/or referenced by dining out        domain model 1622. This domain model represents, among other        things, the idea that dining out involves meal event that occur        at restaurants. The active ontology nodes restaurant 1610 and        meal event 1616 are also included and/or referenced by other        components of the intelligent automated assistant, a shown by        dotted lines in FIG. 8.    -   Active ontologies may include and/or make reference to task flow        models 1086. For example, FIG. 8 depicts an event planning task        flow model 1630, which models the planning of events independent        of domains, applied to a domain-specific kind of event: meal        event 1616. Here, active ontology 1050 includes general event        planning task flow model 1630, which comprises nodes        representing events and other concepts involved in planning        them. Active ontology 1050 also includes the node meal event        1616, which is a particular kind of event. In this example, meal        event 1616 is included or made reference to by both domain model        1622 and task flow model 1630, and both of these models are        included in and/or referenced by active ontology 1050. Again,        meal event 1616 is an example of how active ontologies can unify        elements of various components included and/or referenced by        other components of the intelligent automated assistant, a shown        by dotted lines in FIG. 8.    -   Active ontologies may include and/or make reference to dialog        flow models 1087. For example, FIG. 8 depicts a dialog flow        model 1642 for getting the values of constraints required for a        transaction instantiated on the constraint party size as        represented in concept 1619. Again, active ontology 1050        provides a framework for relating and unifying various        components such as dialog flow models 1087. In this case, dialog        flow model 1642 has a general concept of a constraint that is        instantiated in this particular example to the active ontology        node party size 1619. This particular dialog flow model 1642        operates at the abstraction of constraints, independent of        domain. Active ontology 1050 represents party size property 1619        of party node 1617, which is related to meal event node 1616. In        such an embodiment, intelligent automated assistant 1002 uses        active ontology 1050 to unify the concept of constraint in        dialog flow model 1642 with the property of party size 1619 as        part of a cluster of nodes representing meal event concept 1616,        which is part of the domain model 1622 for dining out.    -   Active ontologies may include and/or make reference to service        models 1088. For example, FIG. 8 depicts a model of a restaurant        reservation service 1672 associated with the dialog flow step        for getting values required for that service to perform a        transaction. In this instance, service model 1672 for a        restaurant reservation service specifies that a reservation        requires a value for party size 1619 (the number of people        sitting at a table to reserve). The concept party size 1619,        which is part of active ontology 1050, also is linked or related        to a general dialog flow model 1642 for asking the user about        the constraints for a transaction; in this instance, the party        size is a required constraint for dialog flow model 1642.    -   Active ontologies may include and/or make reference to domain        entity databases 1072. For example, FIG. 8 depicts a domain        entity database of restaurants 1652 associated with restaurant        node 1610 in active ontology 1050. Active ontology 1050        represents the general concept of restaurant 1610, as may be        used by the various components of intelligent automated        assistant 1002, and it is instantiated by data about specific        restaurants in restaurant database 1652.    -   Active ontologies may include and/or make reference to        vocabulary databases 1058. For example, FIG. 8 depicts a        vocabulary database of cuisines 1662, such as Italian, French,        and the like, and the words associated with each cuisine such as        “French”, “continental”, “provincial”, and the like. Active        ontology 1050 includes restaurant node 1610, which is related to        cuisines served node 1615, which is associated with the        representation of cuisines in cuisines database 1662. A specific        entry in database 1662 for a cuisine, such as “French”, is thus        related through active ontology 1050 as an instance of the        concept of cuisines served 1615.    -   Active ontologies may include and/or make reference to any        database that can be mapped to concepts or other representations        in ontology 1050. Domain entity databases 1072 and vocabulary        databases 1058 are merely two examples of how active ontology        1050 may integrate databases with each other and with other        components of automated assistant 1002. Active ontologies allow        the author, designer, or system builder to specify a nontrivial        mapping between representations in the database and        representations in ontology 1050. For example, the database        schema for restaurants database 1652 may represent a restaurant        as a table of strings and numbers, or as a projection from a        larger database of business, or any other representation        suitable for database 1652. In this example active ontology        1050, restaurant 1610 is a concept node with properties and        relations, organized differently from the database tables. In        this example, nodes of ontology 1050 are associated with        elements of database schemata. The integration of database and        ontology 1050 provides a unified representation for interpreting        and acting on specific data entries in databases in terms of the        larger sets of models and data in active ontology 1050. For        instance, the word “French” may be an entry in cuisines database        1662. Because, in this example, database 1662 is integrated in        active ontology 1050, that same word “French” also has an        interpretation as a possible cuisine served at a restaurant,        which is involved in planning meal events, and this cuisine        serves as a constraint to use when using restaurants reservation        services, and so forth. Active ontologies can thus integrate        databases into the modeling and execution environment to        inter-operate with other components of automated assistant 1002.

As described above, active ontology 1050 allows the author, designer, orsystem builder to integrate components; thus, in the example of FIG. 8,the elements of a component such as constraint in dialog flow model 1642can be identified with elements of other components such as requiredparameter of restaurant reservation service 1672.

Active ontologies 1050 may be embodied as, for example, configurationsof models, databases, and components in which the relationships amongmodels, databases, and components are any of:

-   -   containership and/or inclusion;    -   relationship with links and/or pointers;    -   interface over APis, both internal to a program and between        programs.

For example, referring now to FIG. 9, there is shown an example of analternative embodiment of intelligent automated assistant system 1002,wherein domain models 1056, vocabulary 1058, language patternrecognizers 1060, short term personal memory 1052, and long termpersonal memory 1054 components are organized under a common containerassociated with active ontology 1050, and other components such asactive input elicitation component(s) 1094, language interpreter 1070and dialog flow processor 1080 are associated with active ontology 1050via API relationships.

Active Input Elicitation Component(s) 1094

In at least one embodiment, active input elicitation component(s) 1094(which, as described above, may be implemented in a stand-aloneconfiguration or in a configuration including both server and clientcomponents) may be operable to perform and/or implement various types offunctions, operations, actions, and/or other features such as, forexample, one or more of the following (or combinations thereof):

-   -   Elicit, facilitate and/or process input from the user or the        user's environment, and/or information about their need(s) or        request(s). For example, if the user is looking to find a        restaurant, the input elicitation module may get information        about the user's constraints or preferences for location, time,        cuisine, price, and so forth.    -   Facilitate different kinds of input from various sources, such        as for example, one or more of the following (or combinations        thereof):        -   input from keyboards or any other input device that            generates text        -   input from keyboards in user interfaces that offer dynamic            suggested completions of partial input        -   input from voice or speech input systems        -   input from Graphical User Interfaces (GUUs) in which users            click, select, or otherwise directly manipulate graphical            objects to indicate choices        -   input from other applications that generate text and send it            to the automated assistant, including email, text messaging,            or other text communication platforms

By performing active input elicitation, assistant 1002 is able todisambiguate intent at an early phase of input processing. For example,in an embodiment where input is provided by speech, the waveform mightbe sent to a server 1340 where words are extracted, and semanticinterpretation performed. The results of such semantic interpretationcan then be used to drive active input elicitation, which may offer theuser alternative candidate words to choose among based on their degreeof semantic fit as well as phonetic match.

In at least one embodiment, active input elicitation component(s) 1094actively, automatically, and dynamically guide the user toward inputsthat may be acted upon by one or more of the services offered byembodiments of assistant 1002. Referring now to FIG. 10, there is showna flow diagram depicting a method of operation for active inputelicitation component(s) 1094 according to one embodiment.

The procedure begins 20. In step 21, assistant 1002 may offer interfaceson one or more input channels. For example, a user interface may offerthe user options to speak or type or tap at any stage of aconversational interaction. In step 22, the user selects an inputchannel by initiating input on one modality, such as pressing a buttonto start recording speech or to bring up an interface for typing.

In at least one embodiment, assistant 1002 offers default suggestionsfor the selected modality 23. That is, it offers options 24 that arerelevant in the current context prior to the user entering any input onthat modality. For example, in a text input modality, assistant 1002might offer a list of common words that would begin textual requests orcommands such as, for example, one or more of the following (orcombinations thereof): imperative verbs (e.g., find, buy, reserve, get,call, check, schedule, and the like), nouns (e.g., restaurants, movies,events, businesses, and the like), or menu-like options naming domainsof discourse (e.g., weather, sports, news, and the like).

If the user selects one of the default options in 25, and a preferenceto autosubmit 30 is set, the procedure may return immediately. This issimilar to the operation of a conventional menu selection.

However, the initial option may be taken as a partial input, or the usermay have started to enter a partial input 26. At any point of input, inat least one embodiment, the user may choose to indicate that thepartial input is complete 22, which causes the procedure to return.

In 28, the latest input, whether selected or entered, is added to thecumulative input.

In 29, the system suggestions next possible inputs that are relevantgiven the current input and other sources of constraints on whatconstitutes relevant and/or meaningful input.

In at least one embodiment, the sources of constraints on user input(for example, which are used in steps 23 and 29) are one or more of thevarious models and data sources that may be included in assistant 1002,which may include, but are not limited to, one or more of the following(or combinations thereof):

-   -   Vocabulary 1058. For example, words or phrases that match the        current input may be suggested. In at least one embodiment,        vocabulary may be associated with any or one or more nodes of        active ontologies, domain models, task models, dialog models,        and/or service models.    -   Domain models 1056, which may constrain the inputs that may        instantiate or otherwise be consistent with the domain model.        For example, in at least one embodiment, domain models 1056 may        be used to suggest concepts, relations, properties, and/or        instances that would be consistent with the current input.    -   Language pattern recognizers 1060, which may be used to        recognize idioms, phrases, grammatical constructs, or other        patterns in the current input and be used to suggest completions        that fill out the pattern.    -   Domain entity databases 1072, which may be used to suggest        possible entities in the domain that match the input (e.g.,        business names, movie names, event names, and the like).    -   Short term memory 1052, which may be used to match any prior        input or portion of prior input, and/or any other property or        fact about the history of interaction with a user. For example,        partial input may be matched against cities that the user has        encountered in a session, whether hypothetically (e.g.,        mentioned in queries) and/or physically (e.g., as determined        from location sensors).    -   In at least one embodiment, semantic paraphrases of recent        inputs, request, or results may be matched against the current        input. For example, if the user had previously request “live        music” and obtained concert listing, and then typed “music” in        an active input elicitation environment, suggestions may include        “live music” and/or “concerts”.    -   Long term personal memory 1054, which may be used to suggest        matching items from long term memory. Such matching items may        include, for example, one or more or any combination of: domain        entities that are saved (e.g., “favorite” restaurants, movies,        theaters, venues, and the like), to-do items, list items,        calendar entries, people names in contacts/address books, street        or city names mentioned in contact/address books, and the like.    -   Task flow models 1086, which may be used to suggest inputs based        on the next possible steps of in a task flow.    -   Dialog flow models 1087, which may be used to suggest inputs        based on the next possible steps of in a dialog flow.    -   Service capability models 1088, which may be used to suggest        possible services to employ, by name, category, capability, or        any other property in the model. For example, a user may type        part of the name of a preferred review site, and assistant 1002        may suggest a complete command for querying that review site for        review.

In at least one embodiment, active input elicitation component(s) 1094present to the user a conversational interface, for example, aninterface in which the user and assistant communicate by makingutterances back and forth in a conversational manner. Active inputelicitation component(s) 1094 may be operable to perform and/orimplement various types of conversational interfaces.

In at least one embodiment, active input elicitation component(s) 1094may be operable to perform and/or implement various types ofconversational interfaces in which assistant 1002 uses plies of theconversation to prompt for information from the user according to dialogmodels. Dialog models may represent a procedure for executing a dialog,such as, for example, a series of steps required to elicit theinformation needed to perform a service.

In at least one embodiment, active input elicitation component(s) 1094offer constraints and guidance to the user in real time, while the useris in the midst of typing, speaking, or otherwise creating input. Forexample, active elicitation may guide the user to type text inputs thatare recognizable by an embodiment of assistant 1002 and/or that may beserviced by one or more services offered by embodiments of assistant1002. This is an advantage over passively waiting for unconstrainedinput from a user because it enables the user's efforts to be focused oninputs that may or might be useful, and/or it enables embodiments ofassistant 1002 to apply its interpretations of the input in real time asthe user is inputting it.

At least a portion of the functions, operations, actions, and/or otherfeatures of active input elicitation described herein may beimplemented, at least in part, using various methods and apparatusesdescribed in U.S. patent application Ser. No. 11/518,292 for “Method andApparatus for Building an Intelligent Automated Assistant”, filed Sep.8, 2006.

According to specific embodiments, multiple instances or threads ofactive input elicitation component(s) 1094 may be concurrentlyimplemented and/or initiated via the use of one or more processors 63and/or other combinations of hardware and/or hardware and software.

According to different embodiments, one or more different threads orinstances of active input elicitation component(s) 1094 may be initiatedin response to detection of one or more conditions or events satisfyingone or more different types of minimum threshold criteria for triggeringinitiation of at least one instance of active input elicitationcomponent(s) 1094. Various examples of conditions or events which maytrigger initiation and/or implementation of one or more differentthreads or instances of active input elicitation component(s) 1094 mayinclude, but are not limited to, one or more of the following (orcombinations thereof):

-   -   Start of user session. For example, when the user session starts        up an application that is an embodiment of assistant 1002, the        interface may offer the opportunity for the user to initiate        input, for example, by pressing a button to initiate a speech        input system or clicking on a text field to initiate a text        input session.    -   User input detected.    -   When assistant 1002 explicitly prompts the user for input, as        when it requests a response to a question or offers a menu of        next steps from which to choose.    -   When assistant 1002 is helping the user perform a transaction        and is gathering data for that transaction, e.g., filling in a        form.

In at least one embodiment, a given instance of active input elicitationcomponent(s) 1094 may access and/or utilize information from one or moreassociated databases. In at least one embodiment, at least a portion ofthe database information may be accessed via communication with one ormore local and/or remote memory devices. Examples of different types ofdata which may be accessed by active input elicitation component(s) 1094may include, but are not limited to, one or more of the following (orcombinations thereof):

-   -   database of possible words to use in a textual input;    -   grammar of possible phrases to use in a textual input utterance;    -   database of possible interpretations of speech input;    -   database of previous inputs from a user or from other users;    -   data from any of the various models and data sources that may be        part of embodiments of assistant 1002, which may include, but        are not limited to, one or more of the following (or        combinations thereof):    -   Domain models 1 056;    -   Vocabulary 1058;    -   Language pattern recognizers 1060;    -   Domain entity databases 1 072;    -   Short term memory 1052;    -   Long term personal memory 1054;    -   Task flow models 1 086;    -   Dialog flow models 1087;    -   Service capability models 1088.

According to different embodiments, active input elicitationcomponent(s) 1094 may apply active elicitation procedures to, forexample, one or more of the following (or combinations thereof):

-   -   typed input;    -   speech input;    -   input from graphical user interfaces (GUIs), including gestures;    -   input from suggestions offered in a dialog; and    -   events from the computational and/or sensed environments.

Active Typed Input Elicitation

Referring now to FIG. 11, there is shown a flow diagram depicting amethod for active typed input elicitation according to one embodiment.

The method begins 110. Assistant 1002 receives 111 partial text input,for example via input device 1206. Partial text input may include, forexample, the characters that have been typed so far in a text inputfield. At any time, a user may indicate that the typed input is complete112, as, for example, by pressing an Enter key. If not complete, asuggestion generator generates 114 candidate suggestions 116. Thesesuggestions may be syntactic, semantic, and/or other kinds of suggestionbased any of the sources of information or constraints described herein.If the suggestion is selected 118, the input is transformed 117 toinclude the selected suggestion.

In at least one embodiment, the suggestions may include extensions tothe current input. For example, a suggestion for “rest” may be“restaurants.”

In at least one embodiment, the suggestions may include replacements ofparts of the current input. For example, a suggestion for “rest” may be“places to eat”.

In at least one embodiment, the suggestions may include replacing andrephrasing of parts of the current input. For example, if the currentinput is “find restaurants of style” a suggestion may be “italian” andwhen the suggestion is chosen, the entire input may be rewritten as“find Italian restaurants”.

In at least one embodiment, the resulting input that is returned isannotated 119, so that information about which choices were made in 118is preserved along with the textual input. This enables, for example,the semantic concepts or entities underlying a string to be associatedwith the string when it is returned, which improves accuracy ofsubsequent language interpretation.

Referring now to FIGS. 12 to 21, there are shown screen shotsillustrating some portions of some of the procedures for activetyped-input elicitation according to one embodiment. The screen shotsdepict an example of an embodiment of assistant 1002 as implemented on asmartphone such as the iPhone available from Apple Inc. of Cupertino,Calif. Input is provided to such device via a touch screen, includingon-screen keyboard functionality. One skilled in the art will recognizethat the screen shots depict an embodiment that is merely exemplary, andthat the techniques of the present invention can be implemented on otherdevices and using other layouts and arrangements.

In FIG. 12, screen 1201 includes a top-level set of suggestions 1202shown when no input has been provided in field 1203. This corresponds tono-input step 23 of FIG. 10 applied to step 114 of FIG. 11 where thereis no input.

In FIG. 13, screen 1301 depicts an example of the use of vocabulary tooffer suggested completions 1303 of partial user input 1305 entered infield 1203 using on-screen keyboard 1304. These suggested completions1303 may be part of the function of active input elicitation 1094. Theuser has entered partial user input 1305 including the string “comm.”Vocabulary component 1058 has provided a mapping of this string intothree different kinds of instances, which are listed as suggestedcompletions 1303: the phrase “community & local events” is a category ofthe events domain; “chambers of commerce” is a category of the localbusiness search domain, and “Jewish Community Center” is the name of aninstance of local businesses. Vocabulary component 1058 may provide thedata lookup and management of name spaces like these. The user can tapGo button 1306 to indicate that he or she has finished entering input;this causes assistant 1002 to proceed with the completed text string asa unit of user input.

In FIG. 14, screen 1401 depicts an example in which suggested semanticcompletions 1303 for a partial string “wh” 1305 include entire phraseswith typed parameters. These kinds of suggestions may be enabled by theuse of one or more of the various models and sources of inputconstraints described herein. For example, in one embodiment shown inFIG. 14, “what is happening in city” is an active elicitation of thelocation parameter of the Local Events domain; “where is business name”is an active elicitation of the Business Name constraint of the LocalBusiness Search domain; “what is showing at the venue name” is an activeelicitation of the Venue Name constraint of the Local Events domain; and“what is playing at the movie theater” is an active elicitation of theMovie Theater Name constraint of the Local Events domain. These examplesillustrate that the suggested completions are generated by models ratherthan simply drawn from a database of previously entered queries.

In FIG. 15, screen 1501 depicts a continuation of the same example,after the user has entered additional text 1305 in field 1203. Suggestedcompletions 1303 are updated to match the additional text 1305. In thisexample, data from a domain entity database 1072 were used: venues whosename starts with “f.” Note that this is a significantly smaller and moresemantically relevant set of suggestions than all words that begin with“f.” Again, the suggestions are generated by applying a model, in thiscase the domain model that represents Local Events as happening atVenues, which are Businesses with Names. The suggestions actively elicitinputs that would make potentially meaningful entries when using a LocalEvents service.

In FIG. 16, screen 1601 depicts a continuation of the same example,after the user has selected one of suggested completions 1303. Activeelicitation continues by prompting the user to further specify the typeof information desired, here by presenting a number of specifiers 1602from which the user can select. In this example, these specifiers aregenerated by the domain, task flow, and dialog flow models. The Domainis Local Events, which includes Categories of events that happen onDates in Locations and have Event Names and Feature Performers. In thisembodiment, the fact that these five options are offered to the user isgenerated from the Dialog Flow model that indicates that users should beasked for Constraints that they have not yet entered and from theService Model that indicates that these five Constraints are parametersto Local Event services available to the assistant. Even the choice ofpreferred phrases to use as specifiers, such as “by category” and“featured”, are generated from the Domain Vocabulary databases.

In FIG. 17, screen 1701 depicts a continuation of the same example,after the user has selected one of specifiers 1602.

In FIG. 18, screen 1801 depicts a continuation of the same example,wherein the selected specifier 1602 has been added to field 1203, andadditional specifiers 1602 are presented. The user can select one ofspecifiers 1602 and/or provide additional text input via keyboard 1304.

In FIG. 19, screen 1901 depicts a continuation of the same example,wherein the selected specifier 1602 has been added to field 1203, andyet more specifiers 1602 are presented. In this example, previouslyentered constraints are not actively elicited redundantly.

In FIG. 20, screen 2001 depicts a continuation of the same example,wherein the user has tapped the Go button 1306. The user's input isshown in box 2002, and a message is shown in box 2003, providingfeedback to the user as to the query being performed in response to theuser's input.

In FIG. 21, screen 2101 depicts a continuation of the same example,wherein results have been found. Message is shown in box 2102. Results2103, including input elements allowing the user to view furtherdetails, save the identified event, buy tickets, add notes, or the like.

In one screen 2101, and other displayed screens, are scrollable,allowing the user to scroll upwards to see screen 2001 or otherpreviously presented screens, and to make changes to the query ifdesired.

Active Speech Input Elicitation

Referring now to FIG. 22, there is shown a flow diagram depicting amethod for active input elicitation for voice or speech input accordingto one embodiment.

The method begins 221. Assistant 1002 receives 121 voice or speech inputin the form of an auditory signal. A speech-to-text service 122 orprocessor generates a set of candidate text interpretations 124 of theauditory signal. In one embodiment, speech-to-text service 122 isimplemented using, for example, Nuance Recognizer, available from NuanceCommunications, Inc. of Burlington, Mass.

In one embodiment, assistant 1002 employs statistical language models togenerate candidate text interpretations 124 of speech input 121.

In addition, in one embodiment, the statistical language models aretuned to look for words, names, and phrases that occur in the variousmodels of assistant 1002 shown in FIG. 8. For example, in at least oneembodiment the statistical language models are given words, names, andphrases from some or all of: domain models 1056 (e.g., words and phrasesrelating to restaurant and meal events), task flow models 1086 (e.g.,words and phrases relating to planning an event), dialog flow models1087 (e.g., words and phrases related to the constraints that are neededto gather the inputs for a restaurant reservation), domain entitydatabases 1072 (e.g., names of restaurants), vocabulary databases 1058(e.g., names of cuisines), service models 1088 (e.g., names of serviceprovides such as OpenTable), and/or any words, names, or phrasesassociated with any node of active ontology 1050.

In one embodiment, the statistical language models are also tuned tolook for words, names, and phrases from long-term personal memory 1054.For example, statistical language models can be given text from to-doitems, list items, personal notes, calendar entries, people names incontacts/address books, email addresses, street or city names mentionedin contact/address books, and the like.

A ranking component analyzes the candidate interpretations 124 and ranks126 them according to how well they fit syntactic and/or semantic modelsof intelligent automated assistant 1002. Any sources of constraints onuser input may be used. For example, in one embodiment, assistant 1002may rank the output of the speech-to-text interpreter according to howwell the interpretations parse in a syntactic and/or semantic sense, adomain model, task flow model, and/or dialog model, and/or the like: itevaluates how well various combinations of words in the textinterpretations 124 would fit the concepts, relations, entities, andproperties of active ontology 1050 and its associated models. Forexample, if speech-to-text service 122 generates the two candidateinterpretations “italian food for lunch” and “italian shoes for lunch”,the ranking by semantic relevance 126 might rank “italian food forlunch” higher if it better matches the nodes assistant's 1002 activeontology 1050 (e.g., the words “italian”, “food” and “lunch” all matchnodes in ontology 1050 and they are all connected by relationships inontology 1050, whereas the word “shoes” does not match ontology 1050 ormatches a node that is not part of the dining out domain network).

In various embodiments, algorithms or procedures used by assistant 1002for interpretation of text inputs, including any embodiment of thenatural language processing procedure shown in FIG. 28, can be used torank and score candidate text interpretations 124 generated byspeech-to-text service 122.

In one embodiment, if ranking component 126 determines 128 that thehighest-ranking speech interpretation from interpretations 124 ranksabove a specified threshold, The highest-ranking interpretation may beautomatically selected 130. If no interpretation ranks above a specifiedthreshold, possible candidate interpretations of speech 134 arepresented 132 to the user. The user can then select 136 among thedisplayed choices.

In various embodiments, user selection 136 among the displayed choicescan be achieved by any mode of input, including for example any of themodes of multimodal input described in connection with FIG. 16. Suchinput modes include, without limitation, actively elicited typed input2610, actively elicited speech input 2620, actively presented GUI forinput 2640, and/or the like. In one embodiment, the user can selectamong candidate interpretations 134, for example by tapping or speaking.In the case of speaking, the possible interpretation of the new speechinput is highly constrained by the small set of choices offered 134. Forexample, if offered “Did you mean italian food or italian shoes?” theuser can just say “food” and the assistant can match this to the phrase“italian food” and not get it confused with other global interpretationsof the input.

Whether input is automatically selected 130 or selected 136 by the user,the resulting input 138 is returned. In at least one embodiment, thereturned input is annotated 138, so that information about which choiceswere made in step 136 is preserved along with the textual input. Thisenables, for example, the semantic concepts or entities underlying astring to be associated with the string when it is returned, whichimproves accuracy of subsequent language interpretation. For example, if“Italian food” was offered as one of the candidate interpretations 134based on a semantic interpretation of Cuisine=ItalianFood, then themachine-readable semantic interpretation can be sent along with theuser's selection of the string “Italian food” as annotated text input138.

In at least one embodiment, candidate text interpretations 124 aregenerated based on speech interpretations received as output ofspeech-to-text service 122.

In at least one embodiment, candidate text interpretations 124 aregenerated by paraphrasing speech interpretations in terms of theirsemantic meaning. In some embodiments, there can be multiple paraphrasesof the same speech interpretation, offering different word sense orhomonym alternatives. For example, if speech-to-text service 122indicates “place for meet”, the candidate interpretations presented tothe user could be paraphrased as “place to meet (local businesses)” and“place for meat (restaurants)”.

In at least one embodiment, candidate text interpretations 124 includeoffers to correct substrings.

In at least one embodiment, candidate text interpretations 124 includeoffers to correct substrings of candidate interpretations usingsyntactic and semantic analysis as described herein.

In at least one embodiment, when the user selects a candidateinterpretation, it is returned.

In at least one embodiment, the user is offered an interface to edit theinterpretation before it is returned.

In at least one embodiment, the user is offered an interface to continuewith more voice input before input is returned. This enables one toincrementally build up an input utterance, getting syntactic andsemantic connections, suggestions, and guidance at one iteration.

In at least one embodiment, the user is offered an interface to proceeddirectly from 136 to step 111 of a method of active typed inputelicitation (described above in connection with FIG. 11). This enablesone to interleave typed and spoken input, getting syntactic and semanticcorrections, suggestions, and guidance at one step.

In at least one embodiment, the user is offered an interface to proceeddirectly from step 111 of an embodiment of active typed inputelicitation to an embodiment of active speech input elicitation. Thisenables one to interleave typed and spoken input, getting syntactic andsemantic corrections, suggestions, and guidance at one step.

Active GUI-Based Input Elicitation

Referring now to FIG. 23, there is shown a flow diagram depicting amethod for active input elicitation for GUI-based input according to oneembodiment.

The method begins 140. Assistant 1002 presents 141 graphical userinterface (GUI) on output device 1207, which may include, for example,links and buttons. The user interacts 142 with at least one GUI element.Data 144 is received, and converted 146 to a uniform format. Theconverted data is then returned.

In at least one embodiment, some of the elements of the GUI aregenerated dynamically from the models of the active ontology, ratherthan written into a computer program. For example, assistant 1002 canoffer a set of constraints to guide a restaurant reservation service asregions for tapping on a screen, with each region representing the nameof the constraint and/or a value. For instance, the screen could haverows of a dynamically generated GUI layout with regions for theconstraints Cuisine, Location, and Price Range. If the models of theactive ontology change, the GUI screen would automatically changewithout reprogramming.

Active Dialog Suggestion Input Elicitation

FIG. 24 is a flow diagram depicting a method for active inputelicitation at the level of a dialog flow according to one embodiment.Assistant 1002 suggests 151 possible responses 152. The user selects 154a suggested response. The received input is converted 154 to a uniformformat. The converted data is then returned.

In at least one embodiment, the suggestions offered in step 151 areoffered as follow-up steps in a dialog and/or task flow.

In at least one embodiment, the suggestions offer options to refine aquery, for example using parameters from a domain and/or task model. Forexample, one may be offered to change the assumed location or time of arequest.

In at least one embodiment, the suggestions offer options to chooseamong ambiguous alternative interpretations given by a languageinterpretation procedure or component.

In at least one embodiment, the suggestions offer options to chooseamong ambiguous alternative interpretations given by a languageinterpretation procedure or component.

In at least one embodiment, the suggestions offer options to chooseamong next steps in a workflow associated dialog flow model 1087. Forexample, dialog flow model 1087 may suggest that after gathering theconstrained for one domain (e.g., restaurant dining), assistant 1002should suggest other related domains (e.g., a movie nearby).

Active Monitoring for Relevant Events

In at least one embodiment, asynchronous events may be treated as inputsin an analogous manner to the other modalities of active elicited input.Thus, such events may be provided as inputs to assistant 1002. Onceinterpreted, such events can be treated in a manner similar to any otherinput.

For example, a flight status change may initiate an alert notificationto be sent to a user. If a flight is indicated as being late, assistant1002 may continue the dialog by presenting alternative flights, makingother suggestions, and the like, based on the detected event.

Such events can be of any type. For example, assistant 1002 might detectthat the user just got home, or is lost (off a specified route), or thata stock price hit a threshold value, or that a television show the useris interested in is starting, or that a musician of interest is touringin the area. In any of these situations, assistant 1002 can proceed witha dialog in substantially the same manner as if the user had him- orherself initiated the inquiry. In one embodiment, events can even bebased on data provided from other devices, for example to tell the userwhen a coworker has returned from lunch (the coworker's device cansignal such an event to the user's device, at which time assistant 1002installed on the user's device responds accordingly).

In one embodiment, the events can be notifications or alerts from acalendar, clock, reminder, or to-do application. For example, an alertfrom a calendar application about a dinner date can initiate a dialogwith assistant 1002 about the dining event. The dialog can proceed as ifthe user had just spoken or typed the information about the upcomingdinner event, such as “dinner for 2 in San Francisco”.

In one embodiment, the context of possible event trigger 162 can includeinformation about people, places, times, and other data. These data canbe used as part of the input to assistant 1002 to use in various stepsof processing.

In one embodiment, these data from the context of event trigger 162 canbe used to disambiguate speech or text inputs from the user. Forexample, if a calendar event alert includes the name of a person invitedto the event, that information can help disambiguate input which mightmatch several people with the same or similar name.

Referring now to FIG. 25, there is shown a flow diagram depicting amethod for active monitoring for relevant events according to oneembodiment. In this example, event trigger events are sets of input 162.Assistant 1002 monitors 161 for such events. Detected events may befiltered and sorted 164 for semantic relevance using models, data andinformation available from other components in intelligent automatedassistant 1002. For example, an event that reports a change in flightstatus may be given higher relevance if the short-term or long-termmemory records for a user indicate that the user is on that flightand/or have made inquiries about it to assistant 1002. This sorting andfiltering may then present only the top events for review by the user,who may then choose to pick one or more and act on them.

Event data is converted 166 to a uniform input format, and returned.

In at least one embodiment, assistant 1002 may proactively offerservices associated with events that were suggested for user attention.For example, if a flight status alert indicates a flight may be missed,assistant 1002 may suggest to the user a task flow for replanning theitinerary or booking a hotel.

Input Elicitation Component(s) Example

The following example is intended to help illustrate some of the varioustypes of functions, operations, actions, and/or other features which maybe provided by active input elicitation component(s) 1094. Example:Command completion (what can the user say to assistant 1002?)

The user is facing a text input box with the general instruction toenter “what do you want to do?” Depending on the context and the userinput, any of several system responses are provided. Examples are shownbelow

User Input System Response Comment Case: Null input With an empty fieldFind restaurants, Black text would be and no dialog context, hotels,movies, . . . inserted into the type a question or Book a table, a room,input field. Italicized space character, or a seat, . . . text would bethe enter nothing for Order food, flowers, . . . follow-up suggestionssome period of time. Buy a ticket, a book, once the text was a DVD, . .. inserted. Schedule a meeting, an event, . . . Remember to a thing todo . . . Explain results Case: First word input With empty dialog Findrestaurants, Black text would be context If the user hotels, movies, . .. inserted into the types the first input would be the letter of anyfollow-up suggestions command, it is once the text was suggested. Forinserted example F | Case: Key word input With empty dialog Remember toa thing, Black text would be context If the user to do . . . insertedinto the types the first two Find restaurants by input field. Italicizedletters of any location, cuisine, text would be the keyword, it isfollow-up suggestions suggested as well once the text was as command.For inserted. example F_| Case: Prompting for arguments The user hasentered restaurants by location, Offer not only the or selected acuisine, . . . selection class but “command” verb hotels by location,advertise the con- and nothing else. For availability, . . . straintoptions. example movies by location, Note that name of Find_| genre, . .. place is a just a name of place prompt, and would not insert text.Case: Suggesting criteria The user has already in city, zip code Blacktext would be entered enough to near home, office, inserted into theestablish a task and place name input field. Italicized domain, and nowis named restaurant text would be the prompted for name follow-upsuggestions constraints in order. once the text was find restaurants_|inserted. Case: Adding Criteria The user has said “in” in city, zip codeUntil they enter the and now is prompted Palo Alto, CA next word, thefor locations Menlo Park, CA suggestions are a Find restaurants in_|prompt for what kind The user has said my home of thing to type. The“near” and now my work Places are cities from is prompted for Palo Alto,CA personal memory.. locations Menlo Park, CA The Places are none Findrestaurants or more places from near_| personal memory Case: Addinglocation or other constraints Find restaurants in serving food type orPrompt for other Palo Alto_| cuisine constraints Find restaurants inwith availability tonight, The 7:00 and 9:00 Palo Alto with tomorrow, .. . are our suggestions availability_| described as romantic, based oncurrent time good service tonight tomorrow at 7 pm at 9 pm other time ordate Case: Starting from constraint, unknown task or domain romantic_|restaurants by location, Constraint value from cuisine, . . . ontologytells you hotels by location, which selection availability, . . .classes to use movies by location, genre, . . . comedy_| movies bylocation, “comedy” is a genre, . . . constraint value for events bylocation, . . . genre in movies, a clubs by location genre in events,and “comedy clubs” is a local directory busi- ness category

Example: Name Completion

Here, the user has typed some text without accepting any of thecommands, or he or she is just extending a command with an entity name.The system may attempt to complete the names, depending on context. Italso disambiguates the domain.

Case: words without context User Input System Response Comment Case:words without context il for Il Fornaio (restaurant) May require entityname lookup Ill Forgotten Gains Notice multi-word completion (movie)Show domain as a suggestion. tom cruise movies May require entity nameromantic movies lookup restaurants Based on ontology only Case: namewith context Find restaurants i| Il Fornaio (restaurant) May requireentity name for lookup Uses selection class and location context fromcurrent and past queries to constrain completion. Show domain as asuggestion. tom cruise movies May require entity name lookup Find moviesrom romantic Based on ontology only Roman Holiday (movie) g-rated movies

Example: Choosing Values from a Set

Here, the user is responding to a system request to enter a value for aspecific parameter, such as location, time, cuisine, or genre. The usermay either select from a list or enter a value. As she types, matchingitems from the list are shown as options. Examples are shown below.

Case: valueless selection User Input System Response Comment Restaurantsburgers User may type anything else and serving Ice cream doesn't haveto pick from the menu Hot dogs Restaurants Hamburgers serving h Hot dogsHot sauce Movies playing today tonight Friday night

Example: Reusing Previous Commands

Previous queries are also options to complete on in an autocompleteinterface. They may be just matched as strings (when the input field isempty and there are no known constraints) or they may be suggested asrelevant when in certain situations.

Case: completion on previous queries User Input System Response CommentIntel Italian restaurants Using string matching (normal completion) toretrieve previous Films starring Italian queries actors (recent query)Lunch lunch places in marin (recent query) buy the book naked lunch

Example: Retrieving Personal Memory Items

Assistant 1002 may remember certain events and/or entities in personalmemory associated with the user. Autocomplete can be performed based onsuch remembered items. Examples appear below.

Case: Completion on Events and Entities in Personal Memory

User Input System Response Comment Mary lunch with Mary on Saturday(from My Events) film called “Something about Mary” Lunch lunch withMary on Saturday (from My Events) buy the book naked lunch (from MyTodos) Hob Hobee's Restaurant in Palo Alto (from My Restaurants)

Multimodal Active Input Elicitation

In at least one embodiment, active input elicitation component(s) 1094may process input from a plurality of input modalities. At least onemodality might be implemented with an active input elicitation procedurethat takes advantages of the particular kinds of inputs and methods forselecting from suggested options. A described herein, they may beembodiments of procedures for active input elicitation for text input,speech input, GUI-based input, input in the context of a dialog, and/orinput resulting from event triggers.

In at least one embodiment, for a single instance of intelligentautomated assistant 1002, there may be support for one or more (or anycombination of) typed input, speech input, GUI input, dialog input,and/or event input.

Referring now to FIG. 26, there is shown a flow diagram depicting amethod for multimodal active input elicitation according to oneembodiment. The method begins 100. Inputs may be received concurrentlyfrom one or more or any combination of the input modalities, in anysequence. Thus, the method includes actively eliciting typed input 2610,speech input 2620, GUI-based input 2640, input in the context of adialog 2650, and/or input resulting from event triggers 2660. Any or allof these input sources are unified into unified input format 2690 andreturned. Unified input format 2690 enables the other components ofintelligent automated assistant 1002 to be designed and to operateindependently of the particular modality of the input.

Offering active guidance for multiple modalities and levels enablesconstraint and guidance on the input beyond those available to isolatedmodalities. For example, the kinds of suggestions offered to chooseamong speech, text, and dialog steps are independent, so theircombination is a significant improvement over adding active elicitationtechniques to individual modalities or levels.

Combining multiple sources of constraints as described herein(syntactic/linguistic, vocabulary, entity databases, domain models, taskmodels, service models, and the like) and multiple places where theseconstraints may be actively applied (speech, text, GUI, dialog, andasynchronous events) provides a new level of functionality forhuman-machine interaction.

Domain Models Component(s) 1056

Domain models 1056 component(s) include representations of the concepts,entities, relations, properties, and instances of a domain. For example,dining out domain model 1622 might include the concept of a restaurantas a business with a name and an address and phone number, the conceptof a meal event with a party size and date and time associated with therestaurant.

In at least one embodiment, domain models component(s) 1056 of assistant1002 may be operable to perform and/or implement various types offunctions, operations, actions, and/or other features such as, forexample, one or more of the following (or combinations thereof):

-   -   Domain model component(s) 1056 may be used by automated        assistant 1002 for several processes, including: eliciting input        100, interpreting natural language 200, dispatching to services        400, and generating output 600.    -   Domain model component(s) 1056 may provide lists of words that        might match a domain concept or entity, such as names of        restaurants, which may be used for active elicitation of input        100 and natural language processing 200.    -   Domain model component(s) 1056 may classify candidate words in        processes, for instance, to determine that a word is the name of        a restaurant.    -   Domain model component(s) 1056 may show the relationship between        partial information for interpreting natural language, for        example that cuisine may be associated with business entities        (e.g., “local Mexican food” may be interpreted as “find        restaurants with style=Mexican”, and this inference is possible        because of the information in domain model 1056).    -   Domain model component(s) 1056 may organize information about        services used in service orchestration 1082, for example, that a        particular web service may provide reviews of restaurants.    -   Domain model component(s) 1056 may provide the information for        generating natural language paraphrases and other output        formatting, for example, by providing canonical ways of        describing concepts, relations, properties and instances.

According to specific embodiments, multiple instances or threads of thedomain models component(s) 1056 may be concurrently implemented and/orinitiated via the use of one or more processors 63 and/or othercombinations of hardware and/or hardware and software. For example, inat least some embodiments, various aspects, features, and/orfunctionalities of domain models component(s) 1056 may be performed,implemented and/or initiated by one or more of the following types ofsystems, components, systems, devices, procedures, processes, and thelike (or combinations thereof):

-   -   Domain models component(s) 1056 may be implemented as data        structures that represent concepts, relations, properties, and        instances. These data structures may be stored in memory, files,        or databases.    -   Access to domain model component(s) 1056 may be implemented        through direct APIs, network APIs, database query interfaces,        and/or the like.    -   Creation and maintenance of domain models component(s) 1056 may        be achieved, for example, via direct editing of files, database        transactions, and/or through the use of domain model editing        tools.    -   Domain models component(s) 1056 may be implemented as part of or        in association with active ontologies 1050, which combine models        with instantiations of the models for servers and users.

According to various embodiments, one or more different threads orinstances of domain models component(s) 1056 may be initiated inresponse to detection of one or more conditions or events satisfying oneor more different types of minimum threshold criteria for triggeringinitiation of at least one instance of domain models component(s) 1056.For example, trigger initiation and/or implementation of one or moredifferent threads or instances of domain models component(s) 1056 may betriggered when domain model information is required, including duringinput elicitation, input interpretation, task and domain identification,natural language processing, service orchestration, and/or formattingoutput for users.

In at least one embodiment, a given instance of domain modelscomponent(s) 1056 may access and/or utilize information from one or moreassociated databases. In at least one embodiment, at least a portion ofthe database information may be accessed via communication with one ormore local and/or remote memory devices. For example, data from domainmodel component(s) 1056 may be associated with other model modelingcomponents including vocabulary 1058, language pattern recognizers 1060,dialog flow models 1087, task flow models 1086, service capabilitymodels 1088, domain entity databases 1072, and the like. For example,businesses in domain entity databases 1072 that are classified asrestaurants might be known by type identifiers which are maintained inthe dining out domain model components.

Domain Models Component(s) Example

Referring now to FIG. 27, there is shown a set of screen shotsillustrating an example of various types of functions, operations,actions, and/or other features which may be provided by domain modelscomponent(s) 1056 according to one embodiment.

In at least one embodiment, domain models component(s) 1056 are theunifying data representation that enables the presentation ofinformation shown in screens 103A and 103B about a restaurant, whichcombines data from several distinct data sources and services and whichincludes, for example: name, address, business categories, phone number,identifier for saving to long term personal memory, identifier forsharing over email, reviews from multiple sources, map coordinates,personal notes, and the like.

Language Interpreter Component(s) 1070

In at least one embodiment, language interpreter component(s) 1070 ofassistant 1002 may be operable to perform and/or implement various typesof functions, operations, actions, and/or other features such as, forexample, one or more of the following (or combinations thereof):

-   -   Analyze user input and identify a set of parse results.        -   User input can include any information from the user and            his/her device context that can contribute to understanding            the user's intent, which can include, for example one or            more of the following (or combinations thereof): sequences            of words, the identity of gestures or GUI elements involved            in eliciting the input, current context of the dialog,            current device application and its current data objects,            and/or any other personal dynamic data obtained about the            user such as location, time, and the like. For example, in            one embodiment, user input is in the form of the uniform            annotated input format 2690 resulting from active input            elicitation 1094.        -   Parse results are associations of data in the user input            with concepts, relationships, properties, instances, and/or            other nodes and/or data structures in models, databases,            and/or other representations of user intent and/context.            Parse result associations can be complex mappings from sets            and sequences of words, signals, and other elements of user            input to one or more associated concepts, relations,            properties, instances, other nodes, and/or data structures            described herein.    -   Analyze user input and identify a set of syntactic parse        results, which are parse results that associate data in the user        input with structures that represent syntactic parts of speech,        clauses and phrases including multiword names, sentence        structure, and/or other grammatical graph structures. Syntactic        parse results are described in element 212 of natural language        processing procedure described in connection with FIG. 28.    -   Analyze user input and identify a set of semantic parse results,        which are parse results that associate data in the user input        with structures that represent concepts, relationships,        properties, entities, quantities, propositions, and/or other        representations of meaning and user intent. In one embodiment,        these representations of meaning and intent are represented by        sets of and/or elements of and/or instances of models or        databases and/or nodes in ontologies, as described in element        220 of natural language processing procedure described in        connection with FIG. 28.    -   Disambiguate among alternative syntactic or semantic parse        results as described in element 230 of natural language        processing procedure described in connection with FIG. 28.    -   Determine whether a partially typed input is syntactically        and/or semantically meaningful in an autocomplete procedure such        as one described in connection with FIG. 11.    -   Help generate suggested completions 114 in an autocomplete        procedure such as one described in connection with FIG. 11.    -   Determine whether interpretations of spoken input are        syntactically and/or semantically meaningful in a speech input        procedure such as one described in connection with FIG. 22.

According to specific embodiments, multiple instances or threads oflanguage interpreter component(s) 1070 may be concurrently implementedand/or initiated via the use of one or more processors 63 and/or othercombinations of hardware and/or hardware and software.

According to different embodiments, one or more different threads orinstances of language interpreter component(s) 1070 may be initiated inresponse to detection of one or more conditions or events satisfying oneor more different types of minimum threshold criteria for triggeringinitiation of at least one instance of language interpreter component(s)1070. Various examples of conditions or events which may triggerinitiation and/or implementation of one or more different threads orinstances of language interpreter component(s) 1070 may include, but arenot limited to, one or more of the following (or combinations thereof):

-   -   while eliciting input, including but not limited to        -   Suggesting possible completions of typed input 114 (FIG.            11);        -   Ranking interpretations of speech 126 (FIG. 22);        -   When offering ambiguities as suggested responses in dialog            152 (FIG. 24);    -   when the result of eliciting input is available, including when        input is elicited by any mode of active multimodal input        elicitation 100.

In at least one embodiment, a given instance of language interpretercomponent(s) 1070 may access and/or utilize information from one or moreassociated databases. In at least one embodiment, at least a portion ofsuch data-base information may be accessed via communication with one ormore local and/or remote memory devices. Examples of different types ofdata which may be accessed by the Language Interpreter component(s) mayinclude, but are not limited to, one or more of the following (orcombinations thereof):

-   -   Domain models 1056;    -   Vocabulary 1058;    -   Domain entity databases 1072;    -   Short term memory 1052;    -   Long term personal memory 1054;    -   Task flow models 1086;    -   Dialog flow models 1087;    -   Service capability models 1088.

Referring now also to FIG. 29, there is shown a screen shot illustratingnatural language processing according to one embodiment. The user hasentered (via voice or text) language input 2902 consisting of the phrase“who is playing this weekend at the fillmore.” This phrase is echoedback to the user on screen 2901. Language interpreter component(s) 1070component process input 2902 and generates a parse result. The parseresult associates that input with a request to show the local eventsthat are scheduled for any of the upcoming weekend days at any eventvenue whose name matches “fillmore.” A paraphrase of the parse resultsis shown as 2903 on screen 2901.

Referring now also to FIG. 28, there is shown a flow diagram depictingan example of a method for natural language processing according to oneembodiment.

The method begins 200. Language input 202 is received, such as thestring “who is playing this weekend at the fillmore” in the example ofFIG. 29. In one embodiment, the input is augmented by current contextinformation, such as the current user location and local time. Inword/phrase matching 210, language interpreter component(s) 1070 findassociations between user input and concepts. In this example,associations are found between the string “playing” and the concept oflistings at event venues; the string “this weekend” (along with thecurrent local time of the user) and an instantiation of an approximatetime period that represents the upcoming weekend; and the string“fillmore” with the name of a venue. Word/phrase matching 210 may usedata from, for example, language pattern recognizers 1060, vocabularydatabase 1058, active ontology 1050, short term personal memory 1052,and long term personal memory 1054.

Language interpreter component(s) 1070 generate candidate syntacticparses 212 which include the chosen parse result but may also includeother parse results. For example, other parse results may include thosewherein “playing” is associated with other domains such as games or witha category of event such as sporting events.

Short- and/or long-term memory 1052, 1054 can also be used by languageinterpreter component(s) 1070 in generating candidate syntactic parses212. Thus, input that was provided previously in the same session,and/or known information about the user, can be used, to improveperformance, reduce ambiguity, and reinforce the conversational natureof the interaction. Data from active ontology 1050, domain models 1056,and task flow models 1086 can also be used, to implement evidentialreasoning in determining valid candidate syntactic parses 212.

In semantic matching 220, language interpreter component(s) 1070consider combinations of possible parse results according to how wellthey fit semantic models such as domain models and databases. In thiscase, the parse includes the associations (1) “playing” (a word in theuser input) as “Local Event At Venue” (part of a domain model 1056represented by a cluster of nodes in active ontology 1050) and (2)“fillmore” (another word in the input) as a match to an entity name in adomain entity database 1072 for Local Event Venues, which is representedby a domain model element and active ontology node (Venue Name).

Semantic matching 220 may use data from, for example, active ontology1050, short term personal memory 1052, and long term personal memory1054. For example, semantic matching 220 may use data from previousreferences to venues or local events in the dialog (from short termpersonal memory 1052) or personal favorite venues (from long termpersonal memory 1054).

A set of candidate, or potential, semantic parse results is generated222.

In disambiguation step 230, language interpreter component(s) 1070 weighthe evidential strength of candidate semantic parse results 222. In thisexample, the combination of the parse of “playing” as “Local Event AtVenue” and the match of “fillmore” as a Venme Name is a stronger matchto a domain model than alternative combinations where, for instance,“playing” is associated with a domain model for sports but there is noassociation in the sports domain for “fillmore”.

Disambiguation 230 may use data from, for example, the structure ofactive ontology 1050. In at least one embodiment, the connectionsbetween nodes in an active ontology provide evidential support fordisambiguating among candidate semantic parse results 222. For example,in one embodiment, if three active ontology nodes are semanticallymatched and are all connected in active ontology 1050, this indicateshigher evidential strength of the semantic parse than if these matchingnodes were not connected or connected by longer paths of connections inactive ontology 1050. For example, in one embodiment of semanticmatching 220, the parse that matches both local Event At Venue and VenueName is given increased evidential support because the combinedrepresentations of these aspects of the user intent are connected bylinks and/or relations in active ontology 1050: in this instance, theLocal Event node is connected to the Venue node which is connected tothe Venue Name node which is connected to the entity name in thedatabase of venue names.

In at least one embodiment, the connections between nodes in an activeontology that provide evidential support for disambiguating amongcandidate semantic parse results 222 are directed arcs, forming aninference lattice, in which matching nodes provide evidence for nodes towhich they are connected by directed arcs.

In 232, language interpreter component(s) 1070 sort and select 232 thetop semantic parses as the representation of user intent 290.

Domain Entity Database(s) 1072

In at least one embodiment, domain entity database(s) 1072 may beoperable to perform and/or implement various types of functions,operations, actions, and/or other features such as, for example, one ormore of the following (or combinations thereof):

-   -   Store data about domain entities. Domain entities are things in        the world or computing environment that may be modeled in domain        models. Examples may include, but are not limited to, one or        more of the following (or combinations thereof):        -   Businesses of any kind;        -   Movies, videos, songs and/or other musical products, and/or            any other named entertainment products;        -   Products of any kind;        -   Events;        -   Calendar entries;        -   Cities, states, countries, neighborhoods, and/or other            geographic, geopolitical, and/or geospatial points or            regions;        -   Named places such as landmarks, airports, and the like;    -   Provide database services on these databases, including but not        limited to simple and complex queries, transactions, triggered        events, and the like.

According to specific embodiments, multiple instances or threads ofdomain entity database(s) 1072 may be concurrently implemented and/orinitiated via the use of one or more processors 63 and/or othercombinations of hardware and/or hardware and software. For example, inat least some embodiments, various aspects, features, and/orfunctionalities of domain entity database(s) 1072 may be performed,implemented and/or initiated by database software and/or hardwareresiding on client(s) 1304 and/or on server(s) 1340.

One example of a domain entity database 1072 that can be used inconnection with the present invention according to one embodiment is adatabase of one or more businesses storing, for example, their names andlocations. The database might be used, for example, to look up wordscontained in an input request for matching businesses and/or to look upthe location of a business whose name is known. One skilled in the artwill recognize that many other arrangements and implementations arepossible.

Vocabulary Component(s) 1058

In at least one embodiment, vocabulary component(s) 1058 may be operableto perform and/or implement various types of functions, operations,actions, and/or other features such as, for example, one or more of thefollowing (or combinations thereof):

-   -   Provide databases associating words and strings with concepts,        properties, relations, or instances of domain models or task        models;    -   Vocabulary from vocabulary components may be used by automated        assistant 1002 for several processes, including for example:        eliciting input, interpreting natural language, and generating        output.

According to specific embodiments, multiple instances or threads ofvocabulary component(s) 1058 may be concurrently implemented and/orinitiated via the use of one or more processors 63 and/or othercombinations of hardware and/or hardware and software. For example, inat least some embodiments, various aspects, features, and/orfunctionalities of vocabulary component(s) 1058 may be implemented asdata structures that associate strings with the names of concepts,relations, properties, and instances. These data structures may bestored in memory, files, or databases. Access to vocabulary component(s)1058 may be implemented through direct APIs, network APIs, and/ordatabase query interfaces. Creation and maintenance of vocabularycomponent(s) 1058 may be achieved via direct editing of files, databasetransactions, or through the use of domain model editing tools.Vocabulary component(s) 1058 may be implemented as part of or inassociation with active ontologies 1050. One skilled in the art willrecognize that many other arrangements and implementations are possible.

According to different embodiments, one or more different threads orinstances of vocabulary component(s) 1058 may be initiated in responseto detection of one or more conditions or events satisfying one or moredifferent types of minimum threshold criteria for triggering initiationof at least one instance of vocabulary component(s) 1058. In oneembodiment, vocabulary component(s) 1058 are accessed whenevervocabulary information is required, including, for example, during inputelicitation, input interpretation, and formatting output for users. Oneskilled in the art will recognize that other conditions or events maytrigger initiation and/or implementation of one or more differentthreads or instances of vocabulary component(s) 1058.

In at least one embodiment, a given instance of vocabulary component(s)1058 may access and/or utilize information from one or more associateddatabases. In at least one embodiment, at least a portion of thedatabase information may be accessed via communication with one or morelocal and/or remote memory devices. In one embodiment, vocabularycomponent(s) 1058 may access data from external databases, for instance,from a data warehouse or dictionary.

Language Pattern Recognizer Component(s) 1060

In at least one embodiment, language pattern recognizer component(s)1060 may be operable to perform and/or implement various types offunctions, operations, actions, and/or other features such as, forexample, looking for patterns in language or speech input that indicategrammatical, idiomatic, and/or other composites of input tokens. Thesepatterns correspond to, for example, one or more of the following (orcombinations thereof): words, names, phrases, data, parameters,commands, and/or signals of speech acts.

According to specific embodiments, multiple instances or threads ofpattern recognizer component(s) 1060 may be concurrently implementedand/or initiated via the use of one or more processors 63 and/or othercombinations of hardware and/or hardware and software. For example, inat least some embodiments, various aspects, features, and/orfunctionalities of language pattern recognizer component(s) 1060 may beperformed, implemented and/or initiated by one or more files, databases,and/or programs containing expressions in a pattern matching language.In at least one embodiment, language pattern recognizer component(s)1060 are represented declaratively, rather than as program code; thisenables them to be created and maintained by editors and other toolsother than programming tools. Examples of declarative representationsmay include, but are not limited to, one or more of the following (orcombinations thereof): regular expressions, pattern matching rules,natural language grammars, parsers based on state machines and/or otherparsing models.

One skilled in the art will recognize that other types of systems,components, systems, devices, procedures, processes, and the like (orcombinations thereof) can be used for implementing language patternrecognizer component(s) 1060.

According to different embodiments, one or more different threads orinstances of language pattern recognizer component(s) 1060 may beinitiated in response to detection of one or more conditions or eventssatisfying one or more different types of minimum threshold criteria fortriggering initiation of at least one instance of language patternrecognizer component(s) 1060. Various examples of conditions or eventswhich may trigger initiation and/or implementation of one or moredifferent threads or instances of language pattern recognizercomponent(s) 1060 may include, but are not limited to, one or more ofthe following (or combinations thereof):

-   -   during active elicitation of input, in which the structure of        the language pattern recognizers may constrain and guide the        input from the user;    -   during natural language processing, in which the language        pattern recognizers help interpret input as language;    -   during the identification of tasks and dialogs, in which the        language pattern recognizers may help identify tasks, dialogs,        and/or steps therein.

In at least one embodiment, a given instance of language patternrecognizer component(s) 1060 may access and/or utilize information fromone or more associated databases. In at least one embodiment, at least aportion of the database information may be accessed via communicationwith one or more local and/or remote memory devices. Examples ofdifferent types of data which may be accessed by language patternrecognizer component(s) 1060 may include, but are not limited to, datafrom any of the models various models and data sources that may be partof embodiments of assistant 1002, which may include, but are not limitedto, one or more of the following (or combinations thereof):

-   -   Domain models 1056;    -   Vocabulary 1058;    -   Domain entity databases 1072;    -   Short term memory 1052;    -   Long term personal memory 1054;    -   Task flow models 1086;    -   Dialog flow models 1087;    -   Service capability models 1088.

In one embodiment, access of data from other parts of embodiments ofassistant 1002 may be coordinated by active ontologies 1050.

Referring again to FIG. 14, there is shown an example of some of thevarious types of functions, operations, actions, and/or other featureswhich may be provided by language pattern recognizer component(s) 1060.FIG. 14 illustrates language patterns that language pattern recognizercomponent(s) 1060 may recognize. For example, the idiom “what ishappening” (in a city) may be associated with the task of event planningand the domain of local events.

Dialog Flow Processor Component(s) 1080

In at least one embodiment, dialog flow processor component(s) 1080 maybe operable to perform and/or implement various types of functions,operations, actions, and/or other features such as, for example, one ormore of the following (or combinations thereof):

-   -   Given a representation of the user intent 290 from language        interpretation 200, identify the task a user wants performed        and/or a problem the user wants solved. For example, a task        might be to find a restaurant.    -   For a given problem or task, given a representation of user        intent 290, identify parameters to the task or problem. For        example, the user might be looking for a recommended restaurant        that serves Italian food near the user's home. The constraints        that a restaurant be recommended, serving Italian food, and near        home are parameters to the task of finding a restaurant.    -   Given the task interpretation and current dialog with the user,        such as that which may be represented in personal short term        memory 1052, select an appropriate dialog flow model and        determine a step in the flow model corresponding to the current        state.

According to specific embodiments, multiple instances or threads ofdialog flow processor component(s) 1080 may be concurrently implementedand/or initiated via the use of one or more processors 63 and/or othercombinations of hardware and/or hardware and software.

In at least one embodiment, a given instance of dialog flow processorcomponent(s) 1080 may access and/or utilize information from one or moreassociated databases. In at least one embodiment, at least a portion ofthe database information may be accessed via communication with one ormore local and/or remote memory devices. Examples of different types ofdata which may be accessed by dialog flow processor component(s) 1080may include, but are not limited to, one or more of the following (orcombinations thereof):

-   -   task flow models 1086;    -   domain models 1056;    -   dialog flow models 1087.

Referring now to FIGS. 30 and 31, there are shown screen shotsillustrating an example of various types of functions, operations,actions, and/or other features which may be provided by dialog flowprocessor component(s) according to one embodiment.

As shown in screen 3001, user requests a dinner reservation by providingspeech or text input “book me a table for dinner.” Assistant 1002generates a prompt 3003 asking the user to specify time and party size.

Once these parameters have been provided, screen 3101 is shown.Assistant 1002 outputs a dialog box 3102 indicating that results arebeing presented, and a prompt 3103 asking the user to click a time.Listings 3104 are also displayed.

In one embodiment, such a dialog is implemented as follows. Dialog flowprocessor component(s) 1080 are given a representation of user intentfrom language interpreter component 1070 and determine that theappropriate response is to ask the user for information required toperform the next step in a task flow. In this case, the domain isrestaurants, the task is getting a reservation, and the dialog step isto ask the user for information required to accomplish the next step inthe task flow. This dialog step is exemplified by prompt 3003 of screen3001.

Referring now also to FIG. 32, there is shown a flow diagram depicting amethod of operation for dialog flow processor component(s) 1080according to one embodiment. The flow diagram of FIG. 32 is described inconnection with the example shown in FIGS. 30 and 31.

The method begins 200. Representation of user intent 290 is received. Asdescribed in connection with FIG. 28, in one embodiment, representationof user intent 290 is a set of semantic parses. For the example shown inFIGS. 30 and 31, the domain is restaurants, the verb is “book”associated with restaurant reservations, and the time parameter is theevening of the current day.

In 310, dialog flow processor component(s) 1080 determine whether thisinterpretation of user intent is supported strongly enough to proceed,and/or if it is better supported than alternative ambiguous parses. Inthe current example, the interpretation is strongly supported, with nocompeting ambiguous parses. If, on the other hand, there are competingambiguities or sufficient uncertainty, then step 322 is performed, toset the dialog flow step so that the execution phase causes the dialogto output a prompt for more information from the user.

In 312, the dialog flow processor component(s) 1080 determine thepreferred interpretation of the semantic parse with other information todetermine the task to perform and its parameters. Information may beobtained, for example, from domain models 1056, task flow models 1086,and/or dialog flow models 1087, or any combination thereof. In thecurrent example, the task is identified as getting a reservation, whichinvolves both finding a place that is reservable and available, andeffecting a transaction to reserve a table. Task parameters are the timeconstraint along with others that are inferred in step 312.

In 320, the task flow model is consulted to determine an appropriatenext step. Information may be obtained, for example, from domain models1056, task flow models 1086, and/or dialog flow models 1087, or anycombination thereof. In the example, it is determined that in this taskflow the next step is to elicit missing parameters to an availabilitysearch for restaurants, resulting in prompt 3003 illustrated in FIG. 30,requesting party size and time for a reservation.

As described above, FIG. 31 depicts screen 3101 is shown includingdialog element 3102 that is presented after the user answers the requestfor the party size and reservation time. In one embodiment, screen 3101is presented as the result of another iteration through an automatedcall and response procedure, as described in connection with FIG. 33,which leads to another call to the dialog and flow procedure depicted inFIG. 32. In this instantiation of the dialog and flow procedure, afterreceiving the user preferences, dialog flow processor component(s) 1080determines a different task flow step in step 320: to do an availabilitysearch. When request 390 is constructed, it includes the task parameterssufficient for dialog flow processor component(s) 1080 and servicesorchestration component(s) 1082 to dispatch to a restaurant bookingservice.

Dialog Flow Models Component(s) 1087

In at least one embodiment, dialog flow models component(s) 1087 may beoperable to provide dialog flow models, which represent the steps onetakes in a particular kind of conversation between a user andintelligent automated assistant 1002. For example, the dialog flow forthe generic task of performing a transaction includes steps for gettingthe necessary data for the transaction and confirming the transactionparameters before committing it.

Task Flow Models Component(s) 1086

In at least one embodiment, task flow models component(s) 1086 may beoperable to provide task flow models, which represent the steps onetakes to solve a problem or address a need. For example, the task flowfor getting a dinner reservation involves finding a desirablerestaurant, checking availability, and doing a transaction to get areservation for a specific time with the restaurant.

According to specific embodiments, multiple instances or threads of taskflow models component(s) 1086 may be concurrently implemented and/orinitiated via the use of one or more processors 63 and/or othercombinations of hardware and/or hardware and software. For example, inat least some embodiments, various aspects, features, and/orfunctionalities of task flow models component(s) 1086 may be implementedas programs, state machines, or other ways of identifying an appropriatestep in a flow graph.

In at least one embodiment, task flow models component(s) 1086 may use atask modeling framework called generic tasks. Generic tasks areabstractions that model the steps in a task and their required inputsand generated outputs, without being specific to domains. For example, ageneric task for transactions might include steps for gathering datarequired for the transaction, executing the transaction, and outputtingresults of the transaction—all without reference to any particulartransaction domain or service for implementing it. It might beinstantiated for a domain such as shopping, but it is independent of theshopping domain and might equally well apply to domains of reserving,scheduling, and the like.

At least a portion of the functions, operations, actions, and/or otherfeatures associated with task flow models component(s) 1086 and/orprocedure(s) described herein may be implemented, at least in part,using concepts, features, components, processes, and/or other aspectsdisclosed herein in connection with generic task modeling framework.

Additionally, at least a portion of the functions, operations, actions,and/or other features associated with task flow models component(s) 1086and/or procedure(s) described herein may be implemented, at least inpart, using concepts, features, components, processes, and/or otheraspects relating to constrained selection tasks, as described herein.For example, one embodiment of generic tasks may be implemented using aconstrained selection task model.

In at least one embodiment, a given instance of task flow modelscomponent(s) 1086 may access and/or utilize information from one or moreassociated databases. In at least one embodiment, at least a portion ofthe database information may be accessed via communication with one ormore local and/or remote memory devices. Examples of different types ofdata which may be accessed by task flow models component(s) 1086 mayinclude, but are not limited to, one or more of the following (orcombinations thereof):

-   -   Domain models 1056;    -   Vocabulary 1058;    -   Domain entity databases 1072;    -   Short term memory 1052;    -   Long term personal memory 1054;    -   Dialog flow models 1087;    -   Service capability models 1088.

Referring now to FIG. 34, there is shown a flow diagram depicting anexample of task flow for a constrained selection task 351 according toone embodiment.

Constrained selection is a kind of generic task in which the goal is toselect some item from a set of items in the world based on a set ofconstraints. For example, a constrained selection task 351 may beinstantiated for the domain of restaurants. Constrained selection task351 starts by soliciting criteria and constraints from the user 352. Forexample, the user might be interested in Asian food and may want a placeto eat near his or her office.

In step 353, assistant 1002 presents items that meet the stated criteriaand constraints for the user to browse. In this example, it may be alist of restaurants and their properties which may be used to selectamong them.

In step 354, the user is given an opportunity to refine criteria andconstraints. For example, the user might refine the request by saying“near my office.” The system would then present a new set of results instep 353.

Referring now also to FIG. 35, there is shown an example of screen 3501including list 3502 of items presented by constrained selection task 351according to one embodiment.

In step 355, the user can select among the matching items. Any of anumber of follow-on tasks 359 may then be made available, such as forexample book 356, remember 357, or share 358. In various embodiments,follow-on tasks 359 can involve interaction with web-enabled services,and/or with functionality local to the device (such as setting acalendar appointment, making a telephone call, sending an email or textmessage, setting an alarm, and the like).

In the example of FIG. 35, the user can select an item within list 3502to see more details and to perform additional actions. Referring nowalso to FIG. 36, there is shown an example of screen 3601 after the userhas selected an item from list 3502. Additional information and optionscorresponding to follow-on tasks 359 concerning the selected item aredisplayed.

In various embodiments, the flow steps may be offered to the user in anyof several input modalities, including but not limited to anycombination of explicit dialog prompts and GUI links.

Services Component(s) 1084

Services component(s) 1084 represent the set of services thatintelligent automated assistant 1002 might call on behalf of the user.Any service that can be called may be offered in a services component1084.

In at least one embodiment, services component(s) 1084 may be operableto perform and/or implement various types of functions, operations,actions, and/or other features such as, for example, one or more of thefollowing (or combinations thereof):

-   -   Provide the functions over an API that would normally be        provided by a web-based user interface to a service. For        example, a review website might provide a service API that would        return reviews of a given entity automatically when called by a        program. The API offers to intelligent automated assistant 1002        the services that a human would otherwise obtain by operating        the user interface of the website.    -   Provide the functions over an API that would normally be        provided by a user interface to an application. For example, a        calendar application might provide a service API that would        return calendar entries automatically when called by a program.        The API offers to intelligent automated assistant 1002 the        services that a human would otherwise obtain by operating the        user interface of the application. In one embodiment, assistant        1002 is able to initiate and control any of a number of        different functions available on the device. For example, if        assistant 1002 is installed on a smartphone, personal digital        assistant, tablet computer, or other device, assistant 1002 can        perform functions such as: initiate applications, make calls,        send emails and/or text messages, add calendar events, set        alarms, and the like. In one embodiment, such functions are        activated using services component(s) 1084.    -   Provide services that are not currently implemented in a user        interface, but that are available through an API to assistant in        larger tasks. For example, in one embodiment, an API to take a        street address and return machine-readable geocoordinates might        be used by assistant 1002 as a service component 1084 even if it        has no direct user interface on the web or a device.

According to specific embodiments, multiple instances or threads ofservices component(s) 1084 may be concurrently implemented and/orinitiated via the use of one or more processors 63 and/or othercombinations of hardware and/or hardware and software. For example, inat least some embodiments, various aspects, features, and/orfunctionalities of services component(s) 1084 may be performed,implemented and/or initiated by one or more of the following types ofsystems, components, systems, devices, procedures, processes, and thelike (or combinations thereof):

-   -   implementation of an API exposed by a service, locally or        remotely or any combination;    -   inclusion of a database within automated assistant 1002 or a        database service available to assistant 1002.

For example, a website that offers users an interface for browsingmovies might be used by an embodiment of intelligent automated assistant1002 as a copy of the database used by the website. Servicescomponent(s) 1084 would then offer an internal API to the data, as if itwere provided over a network API, even though the data is kept locally.

As another example, services component(s) 1084 for an intelligentautomated assistant 1002 that helps with restaurant selection and mealplanning might include any or all of the following set of services whichare available from third parties over the network:

-   -   a set of restaurant listing services which lists restaurants        matching name, location, or other constraints;    -   a set of restaurant rating services which return rankings for        named restaurants;    -   a set of restaurant reviews services which returns written        reviews for named restaurants;    -   a geocoding service to locate restaurants on a map;    -   a reservation service that enables programmatic reservation of        tables at restaurants.

Services Orchestration Component(s) 1082

Services orchestration component(s) 1082 of intelligent automatedassistant 1002 executes a service orchestration procedure.

In at least one embodiment, services orchestration component(s) 1082 maybe operable to perform and/or implement various types of functions,operations, actions, and/or other features such as, for example, one ormore of the following (or combinations thereof):

-   -   Dynamically and automatically determine which services may meet        the user's request and/or specified domain(s) and task(s);    -   Dynamically and automatically call multiple services, in any        combination of concurrent and sequential ordering;    -   Dynamically and automatically transform task parameters and        constraints to meet input requirements of service APIs;    -   Dynamically and automatically monitor for and gather results        from multiple services;    -   Dynamically and automatically merge service results data from        various services into to a unified result model;    -   Orchestrate a plurality of services to meet the constraints of a        request;    -   Orchestrate a plurality of services to annotate an existing        result set with auxiliary information;    -   Output the result of calling a plurality of services in a        uniform, service independent representation that unifies the        results from the various services (for example, as a result of        calling several restaurant services that return lists of        restaurants, merge the data on at least one restaurant from the        several services, removing redundancy).

For example, in some situations, there may be several ways to accomplisha particular task. For example, user input such as “remind me to leavefor my meeting across town at 2 pm” specifies an action that can beaccomplished in at least three ways: set alarm clock; create a calendarevent; or call a to-do manager. In one embodiment, servicesorchestration component(s) 1082 makes the determination as to which wayto best satisfy the request.

Services orchestration component(s) 1082 can also make determinations asto which combination of several services would be best to invoke inorder to perform a given overall task. For example, to find and reservea table for dinner, services orchestration component(s) 1082 would makedeterminations as to which services to call in order to perform suchfunctions as looking up reviews, getting availability, and making areservation. Determination of which services to use may depend on any ofa number of different factors. For example, in at least one embodiment,information about reliability, ability of service to handle certaintypes of requests, user feedback, and the like, can be used as factorsin determining which service(s) is/are appropriate to invoke.

According to specific embodiments, multiple instances or threads ofservices orchestration component(s) 1082 may be concurrently implementedand/or initiated via the use of one or more processors and/or othercombinations of hardware and/or hardware and software.

In at least one embodiment, a given instance of services orchestrationcomponent(s) 1082 may use explicit service capability models 1088 torepresent the capabilities and other properties of external services,and reason about these capabilities and properties while achieving thefeatures of services orchestration component(s) 1082. This affordsadvantages over manually programming a set of services that may include,for example, one or more of the following (or combinations thereof):

-   -   Ease of development;    -   Robustness and reliability in execution;    -   The ability to dynamically add and remove services without        disrupting code;    -   The ability to implement general distributed query optimization        algorithms that are driven by the properties and capabilities        rather than hard coded to specific services or APIs.

In at least one embodiment, a given instance of services orchestrationcomponent(s) 1082 may access and/or utilize information from one or moreassociated databases. In at least one embodiment, at least a portion ofthe database information may be accessed via communication with one ormore local and/or remote memory devices. Examples of different types ofdata which may be accessed by services orchestration component(s) 1082may include, but are not limited to, one or more of the following (orcombinations thereof):

-   -   Instantiations of domain models;    -   Syntactic and semantic parses of natural language input;    -   Instantiations of task models (with values for parameters);    -   Dialog and task flow models and/or selected steps within them;    -   Service capability models 1088;    -   Any other information available in an active ontology 1050.

Referring now to FIG. 37, there is shown an example of a procedure forexecuting a service orchestration procedure according to one embodiment.

In this particular example, it is assumed a single user is interestingin finding a good place for dinner at a restaurant, and is engagingintelligent automated assistant 1002 in a conversation to help providethis service.

Consider the task of finding restaurants that are of high quality, arewell reviewed, near a particular location, available for reservation ata particular time, and serve a particular kind of food. These domain andtask parameters are given as input 390.

The method begins 400. At 402, it is determined whether the givenrequest may require any services. In some situations, servicesdelegation may not be required, for example if assistant 1002 is able toperform the desired task itself. For example, in one embodiment,assistant 1002 may be able to answer a factual question without invokingservices delegation. Accordingly, if the request does not requireservices, then standalone flow step is executed in 403 and its result490 is returned. For example, if the task request was to ask forinformation about automated assistant 1002 itself, then the dialogresponse may be handled without invoking any external services.

If, in step 402, it is determined that services delegation is required,services orchestration component(s) 1082 proceed to step 404. In 404,services orchestration component(s) 1082 may match up the taskrequirements with declarative descriptions of the capabilities andproperties of services in service capability models 1088. At least oneservice provider that might support the instantiated operation providesdeclarative, qualitative metadata detailing, for example, one or more ofthe following (or combinations thereof):

-   -   the data fields that are returned with results;    -   which classes of parameters the service provider is statically        known to support;    -   policy functions for parameters the service provider might be        able to support after dynamic inspection of the parameter        values;    -   a performance rating defining how the service performs (e.g.        relational DB, web service, triple store, full-text index, or        some combination thereof);    -   property quality ratings statically defining the expected        quality of property values returned with the result object;    -   an overall quality rating of the results the service may expect        to return.

For example, reasoning about the classes of parameters that service maysupport, a service model may state that services 1, 2, 3, and 4 mayprovide restaurants that are near a particular location (a parameter),services 2 and 3 may filter or rank restaurants by quality (anotherparameter), services 3, 4, and 5 may return reviews for restaurants (adata field returned), service 6 may list the food types served byrestaurants (a data field returned), and service 7 may checkavailability of restaurants for particular time ranges (a parameter).Services 8 through 99 offer capabilities that are not required for thisparticular domain and task.

Using this declarative, qualitative metadata, the task, the taskparameters, and other information available from the runtime environmentof the assistant, services orchestration component(s) 1082 determines404 an optimal set of service providers to invoke. The optimal set ofservice providers may support one or more task parameters (returningresults that satisfy one or more parameters) and also considers theperformance rating of at least one service provider and the overallquality rating of at least one service provider.

The result of step 404 is a dynamically generated list of services tocall for this particular user and request.

In at least one embodiment, services orchestration component(s) 1082considers the reliability of services as well as their ability to answerspecific information requests.

In at least one embodiment, services orchestration component(s) 1082hedges against unreliability by calling overlapping or redundantservices.

In at least one embodiment, services orchestration component(s) 1082considers personal information about the user (from the short termpersonal memory component) to select services. For example, the user mayprefer some rating services over others.

In step 450, services orchestration component(s) 1082 dynamically andautomatically invokes multiple services on behalf of a user. In at leastone embodiment, these are called dynamically while responding to auser's request. According to specific embodiments, multiple instances orthreads of the services may be concurrently called. In at least oneembodiment, these are called over a network using APIs, or over anetwork using web service APIs, or over the Internet using web serviceAPIs, or any combination thereof.

In at least one embodiment, the rate at which services are called isprogrammatically limited and/or managed.

Referring now also to FIG. 38, there is shown an example of a serviceinvocation procedure 450 according to one embodiment. Service invocationis used, for example, to obtain additional information or to performtasks by the use of external services. In one embodiment, requestparameters are transformed as appropriate for the service's API. Onceresults are received from the service, the results are transformed to aresults representation for presentation to the user within assistant1002.

In at least one embodiment, services invoked by service invocationprocedure 450 can be a web service, application running on the device,operating system function, or the like.

Representation of request 390 is provided, including for example taskparameters and the like. For at least one service available from servicecapability models 1088, service invocation procedure 450 performstransformation 452, calling 454, and output-mapping 456 steps.

In transformation step 452, the current task parameters from requestrepresentation 390 are transformed into a form that may be used by atleast one service. Parameters to services, which may be offered as APIsor databases, may differ from the data representation used in taskrequests, and also from at least one other. Accordingly, the objectiveof step 452 is to map at least one task parameter in the one or morecorresponding formats and values in at least one service being called.

For example, the names of businesses such as restaurants may vary acrossservices that deal with such businesses. Accordingly, step 452 wouldinvolve transforming any names into forms that are best suited for atleast one service.

As another example, locations are known at various levels of precisionand using various units and conventions across services. Service I mightmay require ZIP codes, service 2 GPS coordinates, and service 3 postalstreet addresses.

The service is called 454 over an API and its data gathered. In at leastone embodiment, the results are cached. In at least one embodiment, theservices that do not return within a specified level performance (e.g.,as specified in Service Level Agreement or SLA) are dropped.

In output mapping step 456, the data returned by a service is mappedback onto unified result representation 490. This step may includedealing with different formats, units, and so forth.

In step 412, results from multiple services are validated and merged. Inone embodiment, if validated results are collected, an equality policyfunction—defined on a per-domain basis—is then called pair-wise acrossone or more results to determine which results represent identicalconcepts in the real world. When a pair of equal results is discovered,a set of property policy functions—also defined on a per-domainbasis—are used to merge property values into a merged result. Theproperty policy function may use the property quality ratings from theservice capability models, the task parameters, the domain context,and/or the long-term personal memory 1054 to decide the optimal mergingstrategy.

For example, lists of restaurants from different providers ofrestaurants might be merged and duplicates removed. In at least oneembodiment, the criteria for identifying duplicates may include fuzzyname matching, fuzzy location matching, fuzzy matching against multipleproperties of domain entities, such as name, location, phone number,and/or website address, and/or any combination thereof.

In step 414, the results are sorted and trimmed to return a result listof the desired length.

In at least one embodiment, a request relaxation loop is also applied.If, in step 416, services orchestration component(s) 1082 determinesthat the current result list is not sufficient (e.g., it has fewer thanthe desired number of matching items), then task parameters may berelaxed 420 to allow for more results. For example, if the number ofrestaurants of the desired sort found within N miles of the targetlocation is too small, then relaxation would run the request again,looking in an area larger than N miles away, and/or relaxing some otherparameter of the search.

In at least one embodiment, the service orchestration method is appliedin a second pass to “annotate” results with auxiliary data that isuseful to the task.

In step 418, services orchestration component(s) 1082 determines whetherannotation is required. It may be required if, for example, if the taskmay require a plot of the results on a map, but the primary services didnot return geocoordinates required for mapping.

In 422, service capability models 1088 are consulted again to findservices that may return the desired extra information. In oneembodiment, the annotation process determines if additional or betterdata may be annotated to a merged result. It does this by delegating toa property policy function—defined on a per-domain basis—for at leastone property of at least one merged result. The property policy functionmay use the merged property value and property quality rating, theproperty quality ratings of one or more other service providers, thedomain context, and/or the user profile to decide if better data may beobtained. If it is determined that one or more service providers mayannotate one or more properties for a merged result, a cost function isinvoked to determine the optimal set of service providers to annotate.

At least one service provider in the optimal set of annotation serviceproviders is then invoked 450 with the list of merged results, to obtainresults 424. The changes made to at least one merged result by at leastone service provider are tracked during this process, and the changesare then merged using the same property policy function process as wasused in step 412. Their results are merged 426 into the existing resultset.

The resulting data is sorted 428 and unified into a uniformrepresentation 490.

It may be appreciated that one advantage of the methods and systemsdescribed above with respect to services orchestration component(s) 1082is that they may be advantageously applied and/or utilized in variousfields of technology other than those specifically relating tointelligent automated assistants. Examples of such other areas oftechnologies where aspects and/or features of service orchestrationprocedures include, for example, one or more of the following:

-   -   Dynamic “mash ups” on websites and web-based applications and        services;    -   Distributed database query optimization;    -   Dynamic service oriented architecture configuration.

Service Capability Models Component(s) 1088

In at least one embodiment, service capability models component(s) 1088may be operable to perform and/or implement various types of functions,operations, actions, and/or other features such as, for example, one ormore of the following (or combinations thereof):

-   -   Provide machine readable information about the capabilities of        services to perform certain classes of computation;    -   Provide machine readable information about the capabilities of        services to answer certain classes of queries;    -   Provide machine readable information about which classes of        transactions are provided by various services;    -   Provide machine readable information about the parameters to        APIs exposed by various services;    -   Provide machine readable information about the parameters that        may be used in database queries on databases provided by various        services.

Output Processor Component(s) 1090

In at least one embodiment, output processor component(s) 1090 may beoperable to perform and/or implement various types of functions,operations, actions, and/or other features such as, for example, one ormore of the following (or combinations thereof):

-   -   Format output data that is represented in a uniform internal        data structure into forms and layouts that render it        appropriately on different modalities. Output data may include,        for example, communication in natural language between the        intelligent automated assistant and the user; data about domain        entities, such as properties of restaurants, movies, products,        and the like; domain specific data results from information        services, such as weather reports, flight status checks, prices,        and the like; and/or interactive links and buttons that enable        the user to respond by directly interacting with the output        presentation.    -   Render output data for modalities that may include, for example,        any combination of: graphical user interfaces; text messages;        email messages; sounds; animations; and/or speech output.    -   Dynamically render data for different graphical user interface        display engines based on the request. For example, use different        output processing layouts and formats depending on which web        browser and/or device is being used.    -   Render output data in different speech voices dynamically.    -   Dynamically render to specified modalities based on user        preferences.    -   Dynamically render output using user-specific “skins” that        customize the look and feel.    -   Send a stream of output packages to a modality, showing        intermediate status, feedback, or results throughout phases of        interaction with assistant 1002.

According to specific embodiments, multiple instances or threads ofoutput processor component(s) 1090 may be concurrently implementedand/or initiated via the use of one or more processor(s) 63 and/or othercombinations of hardware and/or hardware and software. For example, inat least some embodiments, various aspects, features, and/orfunctionalities of output processor component(s) 1090 may be performed,implemented and/or initiated by one or more of the following types ofsystems, components, systems, devices, procedures, processes, and thelike (or combinations thereof):

-   -   software modules within the client or server of an embodiment of        an intelligent automated assistant;    -   remotely callable services;    -   using a mix of templates and procedural code.

Referring now to FIG. 39, there is shown a flow diagram depicting anexample of a multiphase output procedure according to one embodiment.The multiphase output procedure includes automated assistant 1002processing steps 702 and multiphase output steps 704.

In step 710, a speech input utterance is obtained and a speech-to-textcomponent (such as component described in connection with FIG. 22)interprets the speech to produce a set of candidate speechinterpretations 712. In one embodiment, speech-to-text component isimplemented using, for example, Nuance Recognizer, available from NuanceCommunications, Inc. of Burlington, Mass. Candidate speechinterpretations 712 may be shown to the user in 730, for example inparaphrased form. For example, the interface might show “did you say?”alternatives listing a few possible alternative textual interpretationsof the same speech sound sample.

In at least one embodiment, a user interface is provided to enable theuser to interrupt and choose among the candidate speech interpretations.

In step 714, the candidate speech interpretations 712 are sent to alanguage interpreter 1070, which may produce representations of userintent 716 for at least one candidate speech interpretation 712. In step732, paraphrases of these representations of user intent 716 aregenerated and presented to the user. (See related step 132 of procedure120 in FIG. 22).

In at least one embodiment, the user interface enables the user tointerrupt and choose among the paraphrases of natural languageinterpretations 732.

In step 718, task and dialog analysis is performed. In step 734, taskand domain interpretations are presented to the user using an intentparaphrasing algorithm.

Referring now also to FIG. 40, there is shown a screen shot depicting anexample of output processing according to one embodiment. Screen 4001includes echo 4002 of the user's speech input, generated by step 730.Screen 4001 further includes paraphrase 4003 of the user's intent,generated by step 734. In one embodiment, as depicted in the example ofFIG. 40, special formatting/highlighting is used for key words such as“events”, which may be used to facilitate training of the user forinteraction with intelligent automated assistant 1002. For example, byvisually observing the formatting of the displayed text, the user mayreadily identify and interpret back the intelligent automated assistantrecognizes keywords such as “events”, “next Wednesday”, “San Francisco”,and the like.

Returning to FIG. 39, as requests are dispatched 720 to services andresults are dynamically gathered, intermediate results may be displayedin the form of real-time progress 736. For example, a list ofrestaurants may be returned and then their reviews may be populateddynamically as the results from the reviews services arrive. Servicescan include web-enabled services and/or services that access informationstored locally on the device and/or from any other source.

A uniform representation of response 722 is generated and formatted 724for the appropriate output modality. After the final output format iscompleted, a different kind of paraphrase may be offered in 738. In thisphase, the entire result set may be analyzed and compared against theinitial request. A summary of results or answer to a question may thenbe offered.

Referring also to FIG. 41, there is shown another example of outputprocessing according to one embodiment. Screen 4101 depicts paraphrase4102 of the text interpretation, generated by step 732, real-timeprogress 4103 generated by step 736, and paraphrased summary 7104generated by step 738. Also included are detailed results 4105.

In one embodiment, assistant 1002 is capable of generating output inmultiple modes. Referring now to FIG. 42, there is shown a flow diagramdepicting an example of multimodal output processing according to oneembodiment.

The method begins 600. Output processor 1090 takes uniformrepresentation of response 490 and formats 612 the response according tothe device and modality that is appropriate and applicable. Step 612 mayinclude information from device and modality models 610 and/or domaindata models 614.

Once response 490 has been formatted 612, any of a number of differentoutput mechanisms can be used, in any combination. Examples depicted inFIG. 42 include:

-   -   Generating 620 text message output, which is sent 630 to a text        message channel;    -   Generating 622 email output, which is sent 632 as an email        message;    -   Generating 624 GUI output, which is sent 634 to a device or web        browser for rendering;    -   Generating 626 speech output, which is sent 636 to a speech        generation module.

One skilled in the art will recognize that many other output mechanismscan be used.

In one embodiment, the content of output messages generated bymul-tiphase output procedure 700 is tailored to the mode of multimodaloutput processing 600. For example, if the output modality is speech626, the language of used to paraphrase user input 730, textinterpretations 732, task and domain interpretations 734, progress 736,and/or result summaries 738 may be more or less verbose or use sentencesthat are easier to comprehend in audible form than in written form. Inone embodiment, the language is tailored in the steps of the multiphaseoutput procedure 700; in other embodiments, the multiphase outputprocedure 700 produces an intermediate result that is further refinedinto specific language by multimodal output processing 600.

Short Term Personal Memory Component(s) 1052

In at least one embodiment, short term personal memory component(s) 1052may be operable to perform and/or implement various types of functions,operations, actions, and/or other features such as, for example, one ormore of the following (or combinations thereof):

-   -   Keep a history of the recent dialog between the embodiment of        the assistant and the user, including the history of user inputs        and their interpretations;    -   Keep a history of recent selections by the user in the GUI, such        as which items were opened or explored, which phone numbers were        called, which items were mapped, which movie trailers where        played, and the like;    -   Store the history of the dialog and user interactions in a        database on the client, the server in a user-specific session,        or in client session state such as web browser cookies or RAM        used by the client;    -   Store the list of recent user requests;    -   Store the sequence of results of recent user requests;    -   Store the click-stream history of UI events, including button        presses, taps, gestures, voice activated triggers, and/or any        other user input.    -   Store device sensor data (such as location, time, positional        orientation, motion, light level, sound level, and the like)        which might be correlated with interactions with the assistant.

According to specific embodiments, multiple instances or threads ofshort term personal memory component(s) 1052 may be concurrentlyimplemented and/or initiated via the use of one or more processors 63and/or other combinations of hardware and/or hardware and software.

According to different embodiments, one or more different threads orinstances of short term personal memory component(s) 1052 may beinitiated in response to detection of one or more conditions or eventssatisfying one or more different types of minimum threshold criteria fortriggering initiation of at least one instance of short term personalmemory component(s) 1052. For example, short term personal memorycomponent(s) 1052 may be invoked when there is a user session with theembodiment of assistant 1002, on at least one input form or action bythe user or response by the system.

In at least one embodiment, a given instance of short term personalmemory component(s) 1052 may access and/or utilize information from oneor more associated databases. In at least one embodiment, at least aportion of the database information may be accessed via communicationwith one or more local and/or remote memory devices. For example, shortterm personal memory component(s) 1052 may access data from long-termpersonal memory components(s) 1054 (for example, to obtain user identityand personal preferences) and/or data from the local device about timeand location, which may be included in short term memory entries.

Referring now to FIGS. 43A and 43B, there are shown screen shotsdepicting an example of the use of short term personal memorycomponent(s) 1052 to maintain dialog context while changing location,according to one embodiment. In this example, the user has asked aboutthe local weather, then just says “in new york.” Screen 4301 shows theinitial response, including local weather. When the user says “in newyork”, assistant 1002 uses short term personal memory component(s) 1052to access the dialog context and thereby determine that the currentdomain is weather forecasts. This enables assistant 1002 to interpretthe new utterance “in new york” to mean “what is the weather forecast inNew York this coming?”. Screen 4302 shows the appropriate response,including weather forecasts for New York.

In the example of FIGS. 43A and 43B, what was stored in short termmemory was not only the words of the input “is it going to rain the dayafter tomorrow?” but the system's semantic interpretation of the inputas the weather domain and the time parameter set to the day aftertomorrow.

Long-Term Personal Memory Component(s) 1054

In at least one embodiment, long-term personal memory component(s) 1054may be operable to perform and/or implement various types of functions,operations, actions, and/or other features such as, for example, one ormore of the following (or combinations thereof):

-   -   To persistently store the personal information and data about a        user, including for example his or her preferences, identities,        authentication credentials, accounts, addresses, and the like;    -   To store information that the user has collected by using the        embodiment of assistant 1002, such as the equivalent of        bookmarks, favorites, clippings, and the like;    -   To persistently store saved lists of business entities including        restaurants, hotels, stores, theaters and other venues. In one        embodiment, long-term personal memory component(s) 1054 saves        more than just the names or URLs, but also saves the information        sufficient to bring up a full listing on the entities including        phone numbers, locations on a map, photos, and the like;    -   To persistently store saved movies, videos, music, shows, and        other items of entertainment;    -   To persistently store the user's personal calendar(s), to do        list(s), reminders and alerts, contact databases, social network        lists, and the like;    -   To persistently store shopping lists and wish lists for products        and services, coupons and discount codes acquired, and the like;    -   To persistently store the history and receipts for transactions        including reservations, purchases, tickets to events, and the        like.

According to specific embodiments, multiple instances or threads oflong-term personal memory component(s) 1054 may be concurrentlyimplemented and/or initiated via the use of one or more processors 63and/or other combinations of hardware and/or hardware and software. Forexample, in at least some embodiments, various aspects, features, and/orfunctionalities of long-term personal memory component(s) 1054 may beperformed, implemented and/or initiated using one or more databasesand/or files on (or associated with) clients 1304 and/or servers 1340,and/or residing on storage devices.

According to different embodiments, one or more different threads orinstances of long-term personal memory component(s) 1054 may beinitiated in response to detection of one or more conditions or eventssatisfying one or more different types of minimum threshold criteria fortriggering initiation of at least one instance of long-term personalmemory component(s) 1054. Various examples of conditions or events whichmay trigger initiation and/or implementation of one or more differentthreads or instances of long-term personal memory component(s) 1054 mayinclude, but are not limited to, one or more of the following (orcombinations thereof):

-   -   Long term personal memory entries may be acquired as a side        effect of the user interacting with an embodiment of assistant        1002. Any kind of interaction with the assistant may produce        additions to the long term personal memory, including browsing,        searching, finding, shopping, scheduling, purchasing, reserving,        communicating with other people via an assistant.    -   Long term personal memory may also be accumulated as a        consequence of users signing up for an account or service,        enabling assistant 1002 access to accounts on other services,        using an assistant 1002 service on a client device with access        to other personal information databases such as calendars, to-do        lists, contact lists, and the like.

In at least one embodiment, a given instance of long-term personalmemory component(s) 1054 may access and/or utilize information from oneor more associated databases. In at least one embodiment, at least aportion of the database information may be accessed via communicationwith one or more local and/or remote memory devices, which may belocated, for example, at client(s) 1304 and/or server(s) 1340. Examplesof different types of data which may be accessed by long-term personalmemory component(s) 1054 may include, but are not limited to data fromother personal information databases such as contact or friend lists,calendars, to-do lists, other list managers, personal account and walletmanagers provided by external services 1360, and the like.

Referring now to FIGS. 44A through 44C, there are shown screen shotsdepicting an example of the use of long term personal memorycomponent(s) 1054, according to one embodiment. In the example, afeature is provided (named “My Stuff”), which includes access to savedentities such as restaurants, movies, and businesses that are found viainteractive sessions with an embodiment of assistant 1002. In screen4401 of FIG. 44A, the user has found a restaurant. The user taps on Saveto My Stuff 4402, which saves information about the restaurant inlong-term personal memory component(s) 1054.

Screen 4403 of FIG. 44B depicts user access to My Stuff. In oneembodiment, the user can select among categories to navigate to thedesired item.

Screen 4404 of FIG. 44C depicts the My Restaurant category, includingitems previously stored in My Stuff.

Automated Call and Response Procedure

Referring now to FIG. 33, there is shown a flow diagram depicting anautomatic call and response procedure, according to one embodiment. Theprocedure of FIG. 33 may be implemented in connection with one or moreembodiments of intelligent automated assistant 1002. It may beappreciated that intelligent automated assistant 1002 as depicted inFIG. 1 is merely one example from a wide range of intelligent automatedassistant system embodiments which may be implemented. Other embodimentsof intelligent automated assistant systems (not shown) may includeadditional, fewer and/or different components/features than thoseillustrated, for example, in the example intelligent automated assistant1002 depicted in FIG. 1.

In at least one embodiment, the automated call and response procedure ofFIG. 33 may be operable to perform and/or implement various types offunctions, operations, actions, and/or other features such as, forexample, one or more of the following (or combinations thereof):

-   -   The automated call and response procedure of FIG. 33 may provide        an interface control flow loop of a conversational interface        between the user and intelligent automated assistant 1002. At        least one iteration of the automated call and response procedure        may serve as a ply in the conversation. A conversational        interface is an interface in which the user and assistant 1002        communicate by making utterances back and forth in a        conversational manner.    -   The automated call and response procedure of FIG. 33 may provide        the executive control flow for intelligent automated assistant        1002. That is, the procedure controls the gathering of input,        processing of input, generation of output, and presentation of        output to the user.    -   The automated call and response procedure of FIG. 33 may        coordinate communications among components of intelligent        automated assistant 1002. That is, it may direct where the        output of one component feeds into another, and where the        overall input from the environment and action on the environment        may occur.

In at least some embodiments, portions of the automated call andresponse procedure may also be implemented at other devices and/orsystems of a computer network.

According to specific embodiments, multiple instances or threads of theautomated call and response procedure may be concurrently implementedand/or initiated via the use of one or more processors 63 and/or othercombinations of hardware and/or hardware and software. In at least oneembodiment, one or more or selected portions of the automated call andresponse procedure may be implemented at one or more client(s) 1304, atone or more server(s) 1340, and/or combinations thereof.

For example, in at least some embodiments, various aspects, features,and/or functionalities of the automated call and response procedure maybe performed, implemented and/or initiated by software components,network services, databases, and/or the like, or any combinationthereof.

According to different embodiments, one or more different threads orinstances of the automated call and response procedure may be initiatedin response to detection of one or more conditions or events satisfyingone or more different types of criteria (such as, for example, minimumthreshold criteria) for triggering initiation of at least one instanceof automated call and response procedure. Examples of various types ofconditions or events which may trigger initiation and/or implementationof one or more different threads or instances of the automated call andresponse procedure may include, but are not limited to, one or more ofthe following (or combinations thereof):

-   -   a user session with an instance of intelligent automated        assistant 1002, such as, for example, but not limited to, one or        more of:        -   a mobile device application starting up, for instance, a            mobile device application that is implementing an embodiment            of intelligent automated assistant 1002;        -   a computer application starting up, for instance, an            application that is implementing an embodiment of            intelligent automated assistant 1002;        -   a dedicated button on a mobile device pressed, such as a            “speech input button”;        -   a button on a peripheral device attached to a computer or            mobile device, such as a headset, telephone handset or base            station, a GPS navigation system, consumer appliance, remote            control, or any other device with a button that might be            associated with invoking assistance;        -   a web session started from a web browser to a website            implementing intelligent automated assistant 1002;        -   an interaction started from within an existing web browser            session to a website implementing intelligent automated            assistant 1002, in which, for example, intelligent automated            assistant 1002 service is requested;        -   an email message sent to a modality server 1426 that is            mediating communication with an embodiment of intelligent            automated assistant 1002;        -   a text message is sent to a modality server 1426 that is            mediating communication with an embodiment of intelligent            automated assistant 1002;        -   a phone call is made to a modality server 1434 that is            mediating communication with an embodiment of intelligent            automated assistant 1002;        -   an event such as an alert or notification is sent to an            application that is providing an embodiment of intelligent            automated assistant 1002.    -   when a device that provides intelligent automated assistant 1002        is turned on and/or started.

According to different embodiments, one or more different threads orinstances of the automated call and response procedure may be initiatedand/or implemented manually, automatically, statically, dynamically,concurrently, and/or combinations thereof. Additionally, differentinstances and/or embodiments of the automated call and responseprocedure may be initiated at one or more different time intervals(e.g., during a specific time interval, at regular periodic intervals,at irregular periodic intervals, upon demand, and the like).

In at least one embodiment, a given instance of the automated call andresponse procedure may utilize and/or generate various different typesof data and/or other types of information when performing specific tasksand/or operations. This may include, for example, input data/informationand/or output data/information. For example, in at least one embodiment,at least one instance of the automated call and response procedure mayaccess, process, and/or otherwise utilize information from one or moredifferent types of sources, such as, for example, one or more databases.In at least one embodiment, at least a portion of the databaseinformation may be accessed via communication with one or more localand/or remote memory devices. Additionally, at least one instance of theautomated call and response procedure may generate one or more differenttypes of output data/information, which, for example, may be stored inlocal memory and/or remote memory devices.

In at least one embodiment, initial configuration of a given instance ofthe automated call and response procedure may be performed using one ormore different types of initialization parameters. In at least oneembodiment, at least a portion of the initialization parameters may beaccessed via communication with one or more local and/or remote memorydevices. In at least one embodiment, at least a portion of theinitialization parameters provided to an instance of the automated calland response procedure may correspond to and/or may be derived from theinput data/information.

In the particular example of FIG. 33, it is assumed that a single useris accessing an instance of intelligent automated assistant 1002 over anetwork from a client application with speech input capabilities. Theuser is interested in finding a good place for dinner at a restaurant,and is engaging intelligent automated assistant 1002 in a conversationto help provide this service.

In step 100, the user is prompted to enter a request. The user interfaceof the client offers several modes of input, as described in connectionwith FIG. 26. These may include, for example:

-   -   an interface for typed input, which may invoke an active        typed-input elicitation procedure as illustrated in FIG. 11;    -   an interface for speech input, which may invoke an active speech        input elicitation procedure as illustrated in FIG. 22.    -   an interface for selecting inputs from a menu, which may invoke        active GUI-based input elicitation as illustrated in FIG. 23.

One skilled in the art will recognize that other input modes may beprovided.

In one embodiment, step 100 may include presenting options remainingfrom a previous conversation with assistant 1002, for example using thetechniques described in the active dialog suggestion input elicitationprocedure described in connection with FIG. 24.

For example, by one of the methods of active input elicitation in step100, the user might say to assistant 1002, “where may 1 get some goodItalian around here?” For example, the user might have spoken this intoa speech input component. An embodiment of an active input elicitationcomponent 1094 calls a speech-to-text service, asks the user forconfirmation, and then represents the confirmed user input as a uniformannotated input format 2690.

An embodiment of language interpreter component 1070 is then called instep 200, as described in connection with FIG. 29. Language interpretercomponent 1070 parses the text input and generates a list of possibleinterpretations of the user's intent 290. In one parse, the word“italian” is associated with restaurants of style Italian; “good” isassociated with the recommendation property of restaurants; and “aroundhere” is associated with a location parameter describing a distance froma global sensor reading (for example, the user's location as given byGPS on a mobile device).

In step 300, the representation of the user's intent 290 is passed todialog flow processor 1080, which implements an embodiment of a dialogand flow analysis procedure as described in connection with FIG. 32.Dialog flow processor 1080 determines which interpretation of intent ismost likely, maps this interpretation to instances of domain models andparameters of a task model, and determines the next flow step in adialog flow. In the current example, a restaurant domain model isinstantiated with a constrained selection task to find a restaurant byconstraints (the cuisine style, recommendation level, and proximityconstraints). The dialog flow model indicates that the next step is toget some examples of restaurants meeting these constraints and presentthem to the user.

In step 400, an embodiment of the flow and service orchestrationprocedure 400 is invoked, via services orchestration component 1082. Itinvokes a set of services 1084 on behalf of the user's request to find arestaurant. In one embodiment, these services 1084 contribute some datato a common result. Their data are merged and the resulting list ofrestaurants is represented in a uniform, service-independent form.

In step 500, output processor 1092 generates a dialog summary of theresults, such as, “I found some recommended Italian restaurants nearhere.” Output processor 1092 combines this summary with the outputresult data, and then sends the combination to a module that formats theoutput for the user's particular mobile device in step 600.

In step 700, this device-specific output package is sent to the mobiledevice, and the client software on the device renders it on the screen(or other output device) of the mobile device.

The user browses this presentation, and decides to explore differentoptions. If the user is done 790, the method ends. If the user is notdone 490, another iteration of the loop is initiated by returning tostep 100.

The automatic call and response procedure may be applied, for example toa user's query “how about mexican food?” Such input may be elicited instep 100. In step 200, the input is interpreted as restaurants of styleMexican, and combined with the other state (held in short term personalmemory 1052) to support the interpretation of the same intent as thelast time, with one change in the restaurant style parameter. In step300, this updated intent produces a refinement of the request, which isgiven to service orchestration component(s) 1082 in step 400.

In step 400 the updated request is dispatched to multiple services 1084,resulting in a new set of restaurants which are summarized in dialog in500, formatted for the device in 600, and sent over the network to shownew information on the user's mobile device in step 700.

In this case, the user finds a restaurant of his or her liking, shows iton a map, and sends directions to a friend.

One skilled in the art will recognize that different embodiments of theautomated call and response procedure (not shown) may include additionalfeatures and/or operations than those illustrated in the specificembodiment of FIG. 33, and/or may omit at least a portion of thefeatures and/or operations of automated call and response procedureillustrated in the specific embodiment of FIG. 33.

Constrained Selection

In one embodiment, intelligent automated assistant 1002 uses constrainedselection in its interactions with the user, so as to more effectivelyidentify and present items that are likely to be of interest to theuser.

Constrained selection is a kind of generic task. Generic tasks areabstractions that characterize the kinds of domain objects, inputs,outputs, and control flow that are common among a class of tasks. Aconstrained selection task is performed by selecting items from a choiceset of domain objects (such as restaurants) based on selectionconstraints (such as a desired cuisine or location). In one embodiment,assistant 1002 helps the user explore the space of possible choices,eliciting the user's constraints and preferences, presenting choices,and offering actions to perform on those choices such as to reserve,buy, remember, or share them. The task is complete when the user selectsone or more items on which to perform the action.

Constrained selection is useful in many contexts: for example, picking amovie to see, a restaurant for dinner, a hotel for the night, a place tobuy a book, or the like. In general, constrained selection is usefulwhen one knows the category and needs to select an instance of thecategory with some desired properties.

One conventional approach to constrained selection is a directoryservice. The user picks a category and the system offers a list ofchoices. In a local directory, one may constrain the directory to alocation, such as a city. For instance, in a “yellow pages” service,users select the book for a city and then look up the category, and thebook shows one or more items for that category. The main problem with adirectory service is that the number of possibly relevant choices islarge (e.g., restaurants in a given city).

Another conventional approach is a database application, which providesa way to generate a choice set by eliciting a query from the user,retrieving matching items, and presenting the items in some way thathighlights salient features. The user browses the rows and columns ofthe result set, possibly sorting the results or changing the query untilhe or she finds some suitable candidates. The problem with the databaseservice is that it may require the user to operationalize their humanneed as a formal query and to use the abstract machinery of sort,filter, and browse to explore the resulting data. These are difficultfor most people to do, even with graphical user interfaces.

A third conventional approach is open-ended search, such as “localsearch.” Search is easy to do, but there are several problems withsearch services that make them difficult for people to accomplish thetask of constrained selection. Specifically:

-   -   As with directory search, the user may not just enter a category        and look at one or more possible choice, but must narrow down        the list.    -   If the user can narrow the selection by constraints, it is not        obvious what constraints may be used (e.g., may 1 search for        places that are within walking distance or are open late?)    -   It is not clear how to state constraints (e.g., is it called        cuisine or restaurant type, and what are the possible values?)    -   Multiple preferences conflict; there is usually no objectively        “best” answer to a given situation (e.g., I want a place that is        close by and cheap serving gourmet food with excellent service        and which is open until midnight).    -   Preferences are relative, and they depend on what is available.        For example, if the user may get a table at a highly rated        restaurant, he or she might choose it even though it is        expensive. In general, though, the user would prefer less        expensive options.

In various embodiments, assistant 1002 of the present invention helpsstreamline the task of constrained selection. In various embodiments,assistant 1002 employs database and search services, as well as otherfunctionality, to reduce the effort, on the part of the user, of statingwhat he or she is looking for, considering what is available, anddeciding on a satisfactory solution.

In various embodiments, assistant 1002 helps to make constrainedselection simpler for humans in any of a number of different ways.

For example, in one embodiment, assistant 1002 may operationalizeproperties into constraints. The user states what he or she wants interms of properties of the desired outcome. Assistant 1002operationalizes this input into formal constraints. For example, insteadof saying “find one or more restaurants less than 2 miles from thecenter of Palo Alto whose cuisine includes Italian food” the user mayjust say “Italian restaurants in palo alto.” Assistant 1002 may alsooperationalize qualities requested by the user that are not parametersto a database. For example, if the user requests romantic restaurants,the system may operationalize this as a text search or tag matchingconstraint. In this manner, assistant 1002 helps overcome some of theproblems users may otherwise have with constrained selection. It iseasier, for a user, to imagine and describe a satisfactory solution thanto describe conditions that would distinguish suitable from unsuitablesolutions.

In one embodiment, assistant 1002 may suggest useful selection criteria,and the user need only say which criteria are important at the moment.For example, assistant 1002 may ask “which of these matter: price(cheaper is better), location (closer is better), rating (higher ratedis better)?” Assistant 1002 may also suggest criteria that may requirespecific values; for example, “you can say what kind of cuisine youwould like or a food item you would like”.

In one embodiment, assistant 1002 may help the user make a decisionamong choices that differ on a number of competing criteria (forexample, price, quality, availability, and convenience).

By providing such guidance, assistant 1002 may help users in makingmulti-parametric decisions in any of several ways:

-   -   One is to reduce the dimensionality of the space, combining raw        data such as ratings from multiple sources into a composite        “recommendation” score. The composite score may take into        account domain knowledge about the sources of data (e.g., Zagat        ratings may be more predictive of quality than Yelp).    -   Another approach is to focus on a subset of criteria, turning a        problem of “what are all the possible criteria to consider and        how to they combine?” into a selection of the most important        criteria in a given situation (e.g., “which is more important,        price or proximity?”).    -   Another way to simply the decision making is to assume default        values and preference orders (e.g., all things being equal,        higher rated and closer and cheaper are better). The system may        also remember users' previous responses that indicate their        default values and preferences.    -   Fourth, the system may offer salient properties of items in the        choice set that were not mentioned in the original request. For        example, the user may have asked for local Italian food. The        system may offer a choice set of restaurants, and with them, a        list of popular tags used by reviewers or a tag line from a        guide book (e.g., “a nice spot for a date” “great pasta”). This        could let people pick out a specific item and complete the task.        Research shows that most people make decisions by evaluating        specific instances rather than deciding on criteria and        rationally accepting the one that pops to the top. It also shows        that people learn about features from concrete cases. For        example, when choosing among cars, buyers may not care about        navigation systems until they see that some of the cars have        them (and then the navigation system may become an important        criterion). Assistant 1002 may present salient properties of        listed items that help people pick a winner or that suggest a        dimension along which to optimize.

Conceptual Data Model

In one embodiment, assistant 1002 offers assistance with the constrainedselection task by simplifying the conceptual data model. The conceptualdata model is the abstraction presented to users in the interface ofassistant 1002. To overcome the psychological problems described above,in one embodiment assistant 1002 provides a model that allows users todescribe what they want in terms of a few easily recognized and recalledproperties of suitable choices rather than constraint expressions. Inthis manner, properties can be made easy to compose in natural languagerequests (e.g., adjectives modifying keyword markers) and berecognizable in prompts (“you may also favor recommended restaurants . .. ”). In one embodiment, a data model is used that allows assistant 1002to determine the domain of interest (e.g., restaurants versus hotels)and a general approach to guidance that may be instantiated withdomain-specific properties.

In one embodiment, the conceptual data model used by assistant 1002includes a selection class. This is a representation of the space ofthings from which to choose. For example, in the find-a-restaurantapplication, the selection class is the class of restaurants. Theselection class may be abstract and have subclasses, such as “things todo while in a destination.” In one embodiment, the conceptual data modelassumes that, in a given problem solving situation, the user isinterested in choosing from a single selection class. This assumptionsimplifies the interaction and also allows assistant 1002 to declare itsboundaries of competence (“I know about restaurants, hotels, and movies”as opposed to “I know about life in the city”).

Given a selection class, in one embodiment the data model presented tothe user for the constrained selection task includes, for example:items; item features; selection criteria; and constraints.

Items are instances of the selection class.

Item features are properties, attributes, or computed values that may bepresented and/or associated with at least one item. For example, thename and phone number of a restaurant are item features. Features may beintrinsic (the name or cuisine of a restaurant) or relational (e.g., thedistance from one's current location of interest). They may be static(e.g., restaurant name) or dynamic (rating). They may be compositevalues computed from other data (e.g., a “value for money” score). Itemfeatures are abstractions for the user made by the domain modeler; theydo not need to correspond to underlying data from back-end services.

Selection criteria are item features that may be used to compare thevalue or relevance of items. That is, they are ways to say which itemsare preferred. Selection criteria are modeled as features of the itemsthemselves, whether they are intrinsic properties or computed. Forexample, proximity (defined as distance from the location of interest)is a selection criterion. Location in space-time is a property, not aselection criterion, and it is used along with the location of interestto compute the distance from the location of interest.

Selection criteria may have an inherent preference order. That is, thevalues of any particular criterion may be used to line up items in abest first order. For example, the proximity criterion has an inherentpreference that closer is better. Location, on the other hand, has noinherent preference value. This restriction allows the system to makedefault assumptions and guide the selection if the user only mentionsthe criterion. For example, the user interface might offer to “sort byrating” and assume that higher rated is better.

One or more selection criteria are also item features; they are thosefeatures related to choosing among possible items. However, itemfeatures are not necessarily related to a preference (e.g., the namesand phone numbers of restaurants are usually irrelevant to choosingamong them).

In at least one embodiment, constraints are restrictions on the desiredvalues of the selection criteria. Formally, constraints might berepresented as set membership (e.g., cuisine type includes Italian),pattern matches (e.g., restaurant review text includes “romantic”),fuzzy inequalities (e.g., distance less than a few miles), qualitativethresholds (e.g., highly rated), or more complex functions (e.g., a goodvalue for money). To make things simple enough for normal humans, thisdata model reduces at least one or more constraints to symbolic valuesthat may be matched as words. Time and distance may be excluded fromthis reduction. In one embodiment, the operators and threshold valuesused for implementing constraints are hidden from the user. For example,a constraint on the selection criteria called “cuisine” may berepresented as a symbolic value such as “Italian” or “Chinese.” Aconstraint on rating is “recommended” (a binary choice). For time anddistance, in one embodiment assistant 1002 uses proprietaryrepresentations that handle a range of inputs and constraint values. Forexample, distance might be “walking distance” and time might be“tonight”; in one embodiment, assistant 1002 uses special processing tomatch such input to more precise data.

In at least one embodiment, some constraints may be requiredconstraints. This means that the task simply cannot be completed withoutthis data. For example, it is hard to pick a restaurant without somenotion of desired location, even if one knows the name.

To summarize, a domain is modeled as selection classes with itemfeatures that are important to users. Some of the features are used toselect and order items offered to the user—these features are calledselection criteria. Constraints are symbolic limits on the selectioncriteria that narrow the set of items to those that match.

Often, multiple criteria may compete and constraints may matchpartially. The data model reduces the selection problem from anoptimization (finding the best solution) to a matching problem (findingitems that do well on a set of specified criteria and match a set ofsymbolic constraints). The algorithms for selecting criteria andconstraints and determining an ordering are described in the nextsection.

Methodology for Constrained Selection

In one embodiment, assistant 1002 performs constrained selection bytaking as input an ordered list of criteria, with implicit or explicitconstraints on at least one, and generating a set of candidate itemswith salient features. Computationally, the selection task may becharacterized as a nested search: first, identify a selection class,then identify the important selection criteria, then specify constraints(the boundaries of acceptable solutions), and search through instancesin order of best fit to find acceptable items.

Referring now to FIG. 45, there is shown an example of an abstract model4500 for a constrained selection task as a nested search. In the exampleassistant 1002 identifies 4505 a selection call among all local searchtypes 4501. The identified class is restaurant. Within the set of allrestaurants 4502, assistant 1002 selects 4506 criteria. In the example,the criterion is identified as distance. Within the set of restaurantsin PA 4503, assistant 1002 specifies 4507 constraints for the search. Inthe example, the identified constraint is “Italian cuisine”). Within theset of Italian restaurants in PA 4504, assistant 4508 selects items forpresentation to the user.

In one embodiment, such a nested search is what assistant 1002 does onceit has the relevant input data, rather than the flow for eliciting thedata and presenting results. In one embodiment, such control flow isgoverned via a dialog between assistant 1002 and the user which operatesby other procedures, such as dialog and task flow models. Constrainedselection offers a framework for building dialog and task flow models atthis level of abstraction (that is, suitable for constrained selectiontasks regardless of domain).

Referring now to FIG. 46, there is shown an example of a dialog 4600 tohelp guide the user through a search process, so that the relevant inputdata can be obtained.

In the example dialog 4600, the first step is for the user to state thekind of thing they are looking for, which is the selection class. Forexample, the user might do this by saying “dining in palo alto.” Thisallows assistant 1002 to infer 4601 the task and domain.

Once assistant 1002 has understood the task and domain binding(selection class=restaurants), the next step is to understand whichselection criteria are important to this user, for example by soliciting4603 criteria and/or constraints. In the example above, “in palo alto”indicates a location of interest. In the context of restaurants, thesystem may interpret a location as a proximity constraint (technically,a constraint on the proximity criterion). Assistant 1002 explains whatis needed, receives input. If there is enough information to constrainthe choice set to a reasonable size, then assistant 1002 paraphrases theinput and presents 4605 one or more restaurants that meet the proximityconstraint, sorted in some useful order. The user can then select 4607from this list, or refine 4606 the criteria and constraints. Assistant1002 reasons about the constraints already stated, and usesdomain-specific knowledge to suggest other criteria that might help,soliciting constraints on these criteria as well. For example, assistant1002 may reason that, when recommending restaurants within walkingdistance of a hotel, the useful criteria to solicit would be cuisine andtable availability.

The constrained selection task is complete when the user selects 4607 aninstance of the selection class. In one embodiment, additional follow-ontasks 4602 are enabled by assistant 1002. Thus, assistant 1002 can offerservices that indicate selection while providing some other value.Examples 4608 booking a restaurant, setting a reminder on a calendar,and/or sharing the selection with others by sending an invitation. Forexample, booking a restaurant certainly indicates that it was selected;other options might be to put the restaurant on a calendar or send ininvitation with directions to friends.

Referring now to FIG. 47, there is shown a flow diagram depicting amethod of constrained selection according to one embodiment. In oneembodiment, assistant 1002 operates in an opportunistic andmixed-initiative manner, permitting the user to jump to the inner loop,for instance, by stating task, domain, criteria, and constraints one ormore at once in the input.

The method begins 4701. Input is received 4702 from the user, accordingto any of the modes described herein. If, based on the input, the tasknot known, assistant 1002 requests 4705 clarifying input from the user.

In step 4717, assistant 1002 determines whether the user providesadditional input. If so, assistant 1002 returns to step 4702. Otherwisethe method ends 4799.

If, in step 4703, the task is known, assistant 1002 determines 4704whether the task is constrained selection. If not, assistant 1002proceeds 4706 to the specified task flow.

If, in step 4704, the task is constrained selection, assistant 1002determines 4707 whether the selection class can be determined. If not,assistant 1002 offers 4708 a choice of known selection classes, andreturns to step 4717.

If, in step 4707, the selection class can be determined, assistant 1002determines 4709 whether all required constraints can be determined. Ifnot, assistant 1002 prompts 4710 for required information, and returnsto step 4717.

If, in step 4709, all required constants can be determined, assistant1002 determines 4711 whether any result items can be found, given theconstraints. If there are no items that meet the constraints, assistant1002 offers 4712 ways to relax the constraints. For example, assistant1002 may relax the constraints from lowest to highest precedence, usinga filter/sort algorithm. In one embodiment, if there are items that meetsome of the constraints, then assistant 1002 may paraphrase thesituation (outputting, for example, “I could not find Recommended Greekrestaurants that deliver on Sundays in San Carlos. However, I found 3Greek restaurants and 7 Recommend restaurants in San Carlos.”). In oneembodiment, if there are no items that match any constraints, thenassistant 1002 may paraphrase this situation and prompt for differentconstraints (outputting, for example, “Sorry, I could not find anyrestaurants in Anytown, Tex. You may pick a different location.”).Assistant 1002 returns to step 4717.

If, in step 4711, result items can be found, assistant 1002 offers 4713a list of items. In one embodiment, assistant 1002 paraphrases thecurrently specified criteria and constraints (outputting, for example,“Here are some recommended Italian restaurants in San Jose.”(recommended=yes, cuisine=Italian, proximity=<in San Jose>)). In oneembodiment, assistant 1002 presents a sorted, paginated list of itemsthat meet the known constraints. If an item only shows some of theconstraints, such a condition can be shown as part of the item display.In one embodiment, assistant 1002 offers the user ways to select anitem, for example by initiating another task on that item such asbooking, remembering, scheduling, or sharing. In one embodiment, on anygiven item, assistant 1002 presents item features that are salient forpicking instances of the selection class. In one embodiment, assistant1002 shows how the item meets a constraint; for example, Zagat rating of5 meets the Recommended=yes constraint, and “1 mile away” meets the“within walking distance of an address” constraint. In one embodiment,assistant 1002 allows the user to drill down for more detail on an item,which results in display of more item features.

Assistant 1002 determines 4714 whether the user has selected an item. Ifthe user selects an item, the task is complete. Any follow-on task isperformed 4715, if there is one, and the method ends 4799.

If, in step 4714, the user does not select an item, assistant 1002offers 4716 the user ways to select other criteria and constraints andreturns to step 4717. For example, given the currently specifiedcriteria and constraints, assistant 1002 may offer criteria that aremost likely to constrain the choice set to a desired size. If the userselects a constraint value, that constraint value is added to thepreviously determined constraints when steps 4703 to 4713 are repeated.

Since one or more criteria may have an inherent preference value,selecting the criteria may add information to the request. For example,allowing the user to indicate that positive reviews are valued allowsassistant 1002 to sort by this criterion. Such information can be takeninto account when steps 4703 to 4713 are repeated.

In one embodiment, assistant 1002 allows the user to raise theimportance of a criterion that is already specified, so that it would behigher in the precedence order. For example, if the user asked for fast,cheap, highly recommended restaurants within one block of theirlocation, assistant 1002 may request that the user chooses which ofthese criteria are more important. Such information can be taken intoaccount when steps 4703 to 4713 are repeated.

In one embodiment, the user can provide additional input at any pointwhile the method of FIG. 47 is being performed. In one embodiment,assistant 1002 checks periodically or continuously for such input, and,in response, loops back to step 4703 to process it.

In one embodiment, when outputting an item or list of items, assistant1002 indicates, in the presentation of items, the features that wereused to select and order them. For example, if the user asked for nearbyItalian restaurants, such item features for distance and cuisine may beshown in the presentation of the item. This may include highlightingmatches, as well as listing selection criteria that were involved in thepresentation of an item.

Example Domains

Table 1 provides an example of constrained selection domains that may behandled by assistant 1002 according to various embodiments.

TABLE 1 Based on these criteria special general Select a Location PriceAvailability Type Quality Name Services search search Restaurantproximity affordability open cuisine rating by restaurant delivery menukeywords tables guide, name items review Hotel proximity price rangeavailable motel, rating by hotel name amenities keywords rooms hotel,guide, B&B, . . . review Movie theatre show genre rating by movie titleactors, proximity times review etc Local proximity business rating bybusiness keywords Business category review name Local venue by dateevent title keywords event proximity concert venue by tour music bandname band keywords proximity schedule genre members CD, book, pricerange online, in download, popularity album or artist keywords DVD tostore, etc. physical song name title, etc. buy

Filtering and Sorting Results

In one embodiment, when presenting items that meet currently specifiedcriteria and constraints, a filter/sort methodology can be employed. Inone embodiment selection constraints may serve as both filter and sortparameters to the underlying services. Thus, any selection criterion canbe used to determine which items are in the list, and to compute theorder in which to paginate and show them. Sort order for this task isakin to relevance rank in search. For example, proximity is a criterionwith symbolic constraint values such as “within driving distance” and ageneral notion of sorting by distance. The “driving distance” constraintmight be used to select a group of candidate items. Within that group,closer items might be sorted higher in the list.

In one embodiment, selection constraints and associated filtering andsorting are at discrete “levels”, which are functions of both theunderlying data and the input from the user. For example, proximity isgrouped into levels such as “walking distance”, “taxi distance”,“driving distance.” When sorting, one or more items within walkingdistance are treated as if they were the same distance. The input fromthe user may come into play in the way he or she specifies a constraint.If the user enters “in palo alto”, for example, then one or more itemswithin the Palo Alto city limits are perfect matches and are equivalent.If the user enters, “near the University Avenue train station” then thematch would depend on a distance from that address, with the degree ofmatch dependent on the selection class (e.g., near for restaurants isdifferent than near for hotels). Even within a constraint that may bespecified with a continuous value, a discretization may be applied. Thismay be important for sorting operations, so that multiple criteria mayparticipate in determining the best-first ordering.

In one embodiment, the item list—those items that are considered“matching” or “good enough”—may be shorter or longer than the number ofitems shown on one “page” of the output. Generally, items in the firstpage are given the most attention, but conceptually there is a longerlist, and pagination is simply a function of the form factor of theoutput medium. This means, for instance, that if the user is offered away to sort or browse the items by some criterion, then it is the entireset of items (more than one page worth) that is sorted or browsed.

In one embodiment, there is a precedence ordering among selectioncriteria. That is, some criteria may matter more than others in thefilter and sort. In one embodiment, those criteria selected by the userare given higher precedence than others, and there is a default orderingover one or more criteria. This allows for a general lexicographic sort.The assumption is that there is a meaningful a priori precedence. Forexample, unless the user states otherwise, it may be more important fora restaurant to be close than to be inexpensive. In one embodiment, thea priori precedence ordering is domain-specific. The model allows foruser-specific preferences to override the domain defaults, if that isdesired.

Since the values of constraints can represent several internal datatypes, there are different ways for constraints to match, and they maybe specific to the constraint. For example, in one embodiment:

-   -   Binary constraints match one or more or none. For example,        whether a restaurant is “Fast” might be either true or not.    -   Set membership constraints match one or more or none based on a        property value. For example, cuisine=Greek means the set of        cuisines for a restaurant includes Greek.    -   Enumeration constraints match at a threshold. For example, a        rating criterion might have constraint values rated,        highly-rated, or top-rated. Constraining to highly-rated would        also match top-rated.    -   Numeric constraints match at a threshold that may be criterion        specific. For example, “open late” might be a criterion, and the        user might ask for places open after 10:00 pm. This kind of        constraint may be slightly out of scope for the constrained        selection task, since it is not a symbolic constraint value.        However, in one embodiment, assistant 1002 recognizes some cases        of numeric constraints like this, and maps them to threshold        values with symbolic constraints (e.g., “restaurants in palo        alto open now”→“here are 2 restaurants in palo alto that are        open late”).    -   Location and time are handled specially. A constraint on        proximity might be a location of interest specified at some        level of granularity, and that determines the match. If the user        specifies a city, then city-level matching is appropriate; a ZIP        code may allow for a radius. Assistant 1002 may also understand        locations that are “near” other locations of interest, also        based on special processing. Time is relevant as a constraint        value of criteria that have threshold value based on a service        call, such as table availability or flights within a given time        range.

In one embodiment, constraints can be modeled so that there is a singlethreshold value for selection and a small set of discrete values forsorting. For example, the affordability criterion might be modeled as aroughly binary constraint, where affordable restaurants are any undersome threshold price range. When the data justify multiple discretelevels for selection, constraints can be modeled using a gradient ofmatching. In one embodiment two levels of matching (such as strong andweak matching) may be provided; however, one skilled in the art willrecognize that in other embodiments, any number of levels of matchingcan be provided. For example, proximity may be matched with a fuzzyboundary, so that things that are near the location of interest maymatch weakly. The operational consequence of a strong or weak match isin the filter/sort algorithm as described below.

For at least one criterion, an approach to matching and defaultthresholds can be established, if relevant. The user may be able to sayjust the name of the constraint, a symbolic constraint value, or aprecise constraint expression if it is handled specially (such as timeand location).

An ideal situation for constrained selection occurs when the user statesconstraints that result in a short list of candidates, one or more ofwhich meet the constraints. The user then chooses among winners based onitem features. In many cases, however, the problem is over- orunder-constrained. When it is over-constrained, there are few or noitems that meet the constraints. When it is under-constrained, there areso many candidates that examining the list is not expedient. In oneembodiment, the general constrained selection model of the presentinvention is able to handle multiple constraints with robust matchingand usually produce something to choose from. Then the user may elect torefine their criteria and constraints or just complete the task with a“good enough” solution.

Method

In one embodiment, the following method is used for filtering andsorting results:

1. Given an ordered list of selection criteria selected by the user,determine constraints on at least one.

-   -   a. If the user specified a constraint value, use it. For        example, if the user said “greek food” the constraint is        cuisine=Greek. If the user said “san Francisco” the constraint        is In the City of San Francisco. If the user said “south of        market” then the constraint is In the Neighborhood of SoMa.    -   b. Otherwise use a domain- and criteria-specific default. For        example, if the user said “a table at some thai place” he or she        is indicating that the availability criterion is relevant, but        he or she did not specify a constraint value. The default        constraint values for availability might be some range of date        times such as tonight and a default party size of 2.

2. Select a minimum of N results by specified constraints.

-   -   a. Try to get N results at strong match.    -   b. If that fails, try to relax constraints, in reverse        precedence order. That is, match at strong level for one or more        of the criteria except the last, which may match at a weak        level. If there is no weak match for that constraint, then try        weak matches up the line from lowest to highest precedence.    -   c. Then repeat the loop allowing failure to match on        constraints, from lowest to highest precedence.

3. After getting a minimum choice set, sort lexicographically over oneor more criteria (which may include user-specified criteria as well asother criteria) in precedence order.

-   -   a. Consider the set of user-specified criteria as highest        precedence, then one or more remaining criteria in their a        priori precedence. For example, if the a priori precedence is        (availability, cuisine, proximity, rating), and the user gives        constraints on proximity and cuisine, then the sort precedence        is (cuisine, proximity, availability, rating).    -   b. Sort on criteria using discrete match levels (strong, weak,        none), using the same approach as in relaxing constraints, this        time applied the full criteria list.        -   i. If a choice set was obtained without relaxing            constraints, then one or more of the choice set may “tie” in            the sort because they one or more match at strong levels.            Then, the next criteria in the precedence list may kick in            to sort them. For example, if the user says cuisine=Italian,            proximity=In San Francisco, and the sort precedence is            (cuisine, proximity, availability, rating), then one or more            the places on the list have equal match values for cuisine            and proximity. So the list would be sorted on availability            (places with tables available bubble to the top). Within the            available places, the highest rated ones would be at the            top.        -   ii. If the choice set was obtained by relaxing constraints,            then one or more of the fully matching items are at the top            of the list, then the partially matching items. Within the            matching group, they are sorted by the remaining criteria,            and the same for the partially matching group. For example,            if there were only two Italian restaurants in San Francisco,            then the available one would be shown first, then the            unavailable one. Then the rest of the restaurants in San            Francisco would be shown, sorted by availability and rating.

Precedence Ordering

The techniques described herein allow assistant 1002 to be extremelyrobust in the face of partially specified constraints and incompletedata. In one embodiment, assistant 1002 uses these techniques togenerate a user list of items in best-first order, i.e. according torelevance.

In one embodiment, such relevance sorting is based on an a prioriprecedence ordering. That is, of the things that matter about a domain,a set of criteria is chosen and placed in order of importance. One ormore things being equal, criteria higher in the precedence order may bemore relevant to a constrained selection among items than those lower inthe order. Assistant 1002 may operate on any number of criteria. Inaddition, criteria may be modified over time without breaking existingbehaviors.

In one embodiment, the precedence order among criteria may be tuned withdomain-specific parameters, since the way criteria interact may dependon the selection class. For example, when selecting among hotels,availability and price may be dominant constraints, whereas forrestaurants, cuisine and proximity may be more important.

In one embodiment, the user may override the default criteria orderingin the dialog. This allows the system to guide the user when searchesare over-constrained, by using the ordering to determine whichconstraints should be relaxed. For example, if the user gave constraintson cuisine, proximity, recommendation, and food item, and there were nofully matching items, the user could say that food item was moreimportant than recommendation level and change the mix so the desiredfood item matches were sorted to the top.

In one embodiment, when precedence order is determined, user-specifiedconstraints take precedence over others. For example, in one embodiment,proximity is a required constraint and so is always specified, andfurther has precedence over other unselected constraints. Therefore itdoes not have to be the highest precedence constraint in order to befairly dominant. Also, many criteria may not match at one or more unlessa constraint is given by the user, and so the precedence of thesecriteria only matters within user-selected criteria.

For example, when the user specifies a cuisine it is important to them,and otherwise is not relevant to sorting items.

For example, the following is a candidate precedence sorting paradigmfor the restaurant domain:

-   -   1. cuisine* (not sortable unless a constraint value is given)    -   2. availability* (sortable using a default constraint value,        e.g., time)    -   3. recommended    -   4. proximity* (a constraint value is always given)    -   5. affordability    -   6. may deliver    -   7. food item (not sortable unless a constraint value, e.g., a        keyword, is given)    -   8. keywords (not sortable unless a constraint value, e.g., a        keyword, is given)    -   9. restaurant name

The following is an example of a design rationale for the above sortingparadigm:

-   -   If a user specifies a cuisine, he or she wants it to stick.    -   One or more things being equal, sort by rating level (it is the        highest precedence among criteria than may be used to sort        without a constraint).    -   In at least one embodiment, proximity may be more important than        most things. However, since it matches at discrete levels (in a        city, within a radius for walking and the like), and it is        always specified, then most of the time most matching items may        “tie” on proximity.    -   Availability (as determined by a search on a website such as        open-table.com, for instance) is a valuable sort criterion, and        may be based on a default value for sorting when not specified.        If the user indicates a time for booking, then only available        places may be in the list and the sort may be based on        recommendation.    -   If the user says they want highly recommended places, then it        may sort above proximity and availability, and these criteria        may be relaxed before recommendation. The assumption is that if        someone is looking for nice place, they may be willing to drive        a bit farther and it is more important than a default table        availability. If a specific time for availability is specified,        and the user requests recommended places, then places that are        both recommended and available may come first, and        recommendation may relax to a weak match before availability        fails to match at one or more.    -   The remaining constraints except for name are one or more based        on incomplete data or matching. So they are weak sort heuristics        by default, and when they are specified the match one or        more-or-none.    -   Name may be used as a constraint to handle the case where        someone mentions the restaurant by name, e.g., find one or more        Hobee's restaurants near Palo Alto. In this case, one or more        items may match the name, and may be sorted by proximity (the        other specified constraint in this example).

Domain Modeling: Mapping Selection Criteria to Underlying Data

It may be desirable to distinguish between the data that are availablefor computation by assistant 1002 and the data used for makingselections. In one embodiment, assistant 1002 uses a data model thatreduces the complexity for the user by folding one or more kinds of dataused to distinguish among items into a simple selection criteria model.Internally, these data may take several forms. Instances of theselection class can have intrinsic properties and attributes (such ascuisine of a restaurant), may be compared along dimensions (such as thedistance from some location), and may be discovered by some query (suchas whether it matches a text pattern or is available at a given time).They may also be computed from other data which are not exposed to theuser as selection criteria (e.g., weighted combinations of ratings frommultiple sources). These data are one or more relevant to the task, butthe distinctions among these three kinds of data are not relevant to theuser. Since the user thinks in terms of features of the desired choicerather than in properties and dimensions, assistant 1002 operationalizesthese various criteria into features of the items. Assistant 1002provides a user-facing domain data model and maps it to data found inweb services.

One type of mapping is an isomorphism from underlying data touser-facing criteria. For example, the availability of tables forreservations as seen by the user could be exactly what an onlinereservation website, such as opentable.com, offers, using the samegranularity for time and party size.

Another type of mapping is a normalization of data from one or moreservices to a common value set, possibly with a unification ofequivalent values. For example, cuisines of one or more restaurants maybe represented as a single ontology in assistant 1002, and mapped tovarious vocabularies used in different services. That ontology might behierarchical, and have leaf nodes pointing to specific values from atleast one service. For example, one service might have a cuisine valuefor “Chinese”, another for “Szechuan”, and a third for “Asian.” Theontology used by assistant 1002 would cause references to “Chinese food”or “Szechuan” to semantically match one or more of these nodes, withconfidence levels reflecting the degree of match.

Normalization might also be involved when resolving differences inprecision. For example, the location of a restaurant may be given to thestreet level in one service but only to city in another. In oneembodiment, assistant 1002 uses a deep structural representation oflocations and times that may be mapped to different surface data values.

In one embodiment, assistant 1002 uses a special kind of mapping foropen-ended qualifiers (e.g., romantic, quiet) which may be mapped tomatches in full text search, tags, or other open-textured features. Thename of the selection constraint in this case would be something like“is described as”.

In at least one embodiment, constraints may be mapped to operationalpreference orderings. That is, given the name of a selection criterionand its constraint value, assistant 1002 is able to interpret thecriterion as an ordering over possible items. There are severaltechnical issues to address in such a mapping. For example:

-   -   Preference orderings may conflict. The ordering given by one        constraint may be inconsistent or even inversely correlated with        the ordering given by another. For example, price and quality        tend to be in opposition. In one embodiment, assistant 1002        interprets constraints chosen by the user in a weighted or        otherwise combined ordering that reflects the user's desires but        is true to the data. For example, the user may ask for “cheap        fast food French restaurants within walking distance rated        highly.” In many locations, there may not be any such        restaurant. However, in one embodiment, assistant 1002 may show        a list of items that tries to optimize for at least one        constraint, and explain why at least one is listed. For example,        item one might be “highly rated French cuisine” and another        “cheap fast food within walking distance”.    -   Data may be used as either hard or soft constraints. For        example, the price range of a restaurant may be important to        choosing one, but it may be difficult to state a threshold value        for price up-front. Even seemingly hard constraints like cuisine        may be, in practice, soft constraints because of partial        matching. Since, in one embodiment, assistant 1002 using a data        modeling strategy that seeks to flatten one or more criteria        into symbolic values (such as “cheap” or “close”), these        constraints may be mapped into a function that gets the criteria        and order right, without being strict about matching specific        threshold values. For symbolic criteria with clear objective        truth values, assistant 1002 may weigh the objective criteria        higher than other criteria, and make it clear in the explanation        that it knows that some of the items do not strictly match the        requested criteria.    -   Items may match some but not one or more constraints, and the        “best fitting” items may be shown.    -   In general, assistant 1002 determines which item features are        salient for a domain, and which may serve as selection criteria,        and for at least one criteria, possible constraint values. Such        information can be provided, for example, via operational data        and API calls.

Paraphrase and Prompt Text

As described above, in one embodiment assistant 1002 provides feedbackto show it understands the user's intent and is working toward theuser's goal by producing paraphrases of its current understanding. Inthe conversational dialog model of the present invention, the paraphraseis what assistant 1002 outputs after the user's input, as a preface (forexample, paraphrase 4003 in FIG. 40) or summary of the results to follow(for example, list 3502 in FIG. 35).

The prompt is a suggestion to the user about what else they can do torefine their request or explore the selection space along somedimensions.

In one embodiment, the purposes of paraphrase and prompt text include,for example:

-   -   to show that assistant 1002 understands the concepts in the        user's input, not just the text;    -   to indicate the boundaries of assistant's 1002 understanding;    -   to guide the user to enter text that is required for the assumed        task;    -   to help the user explore the space of possibilities in        constrained selection;    -   to explain the current results obtained from services in terms        of the user's stated criteria and assistant's 1002 assumptions        (for example, to explain the results of under- and        over-constrained requests).

For example, the following paraphrase and prompt illustrates several ofthese goals:

-   -   User input: indonesian food in menlo park    -   System interpretation:    -   Task=constrainedSelection    -   SelectionClass=restaurant    -   Constraints:        -   Location=Menlo Park, Calif.        -   Cuisine=Indonesian (known in ontology)    -   Results from Services: no strong matches    -   Paraphrase: Sorry, I can't find any Indonesian restaurants near        Menlo Park.    -   Prompt: You could try other cuisines or locations.    -   Prompt under hypertext links:    -   Indonesian: You can try other food categories such as Chinese,        or a favorite food item such as steak.    -   Menlo Park: Enter a location such as a city, neighborhood,        street address, or “near” followed by a landmark.    -   Cuisines: Enter a food category such as Chinese or Pizza.    -   Locations: Enter a location: a city, zip code, or “near”        followed by the name of a place.

In one embodiment, assistant 1002 responds to user input relativelyquickly with the paraphrase. The paraphrase is then updated afterresults are known. For example, an initial response may be “Looking forIndonesian restaurants near Menlo Park . . . ” Once results areobtained, assistant 1002 would update the text to read, “Sorry, I can'tfind any Indonesian restaurants near Menlo Park. You could try othercuisines or locations.” Note that certain items are highlighted(indicated here by underline), indicating that those items representconstraints that can be relaxed or changed.

In one embodiment, special formatting/highlighting is used for key wordsin the paraphrase. This can be helpful to facilitate training of theuser for interaction with intelligent automated assistant 1002, byindicating to the user which words are most important to, and morelikely to be recognized by, assistant 1002. User may then be more likelyto use such words in the future.

In one embodiment, paraphrase and prompt are generated using anyrelevant context data. For example, any of the following data items canbe used, alone or in combination:

-   -   The parse—a tree of ontology nodes bound to their matching input        tokens, with annotations and exceptions. For each node in the        parse, this may include the node's metadata and/or any tokens in        the input that provide evidence for the node's value.    -   The task, if known    -   The selection class.    -   The location constraint, independent of selection class.    -   Which required parameters are unknown for the given selection        class (e.g., location is a required constraint on restaurants).    -   The name of a named entity in the parse that is an instance of        the selection class, if there is one (e.g., a specific        restaurant or movie name.)    -   Is this a follow-up refinement or the beginning of a        conversation? (Reset starts a new conversation.)    -   Which constraints in the parse are bound to values in the input        that changed their values? In other words, which constraints        were just changed by the latest input?    -   Is the selection class inferred or directly stated?    -   Sorted by quality, relevance, or proximity?    -   For each constraint specified, how well was it matched?    -   Was refinement entered as text or clicking?

In one embodiment, the paraphrase algorithm accounts for the query,domain model 1056, and the service results. Domain model 1056 containsclasses and features including metadata that is used to decide how togenerate text. Examples of such metadata for paraphrase generationinclude:

-   -   OsConstraint={trueIfalse}    -   IsMultiValued={trueIfalse}    -   ConstraintType={EntityName, Location, Time, CategoryConstraint,        AvailabilityConstraint, BinaryConstraint, SearchQualifier,        Guessed-Qualifier}    -   DisplayName=string    -   DisplayTemplateSingular=string    -   DisplayTemplatePlural=string    -   Grammatical Role={AdjectiveBeforeNoun,Noun,ThatClauseModifer}

For example, a parse might contain these elements:

-   -   Class: Restaurant    -   IsConstraint=false    -   DisplayTemplateSingular=“restaurant”    -   DisplayTemplatePlural=“restaurants”    -   GrammaticalRole=Noun    -   Feature: RestaurantName (example: “Il Fornaio”)    -   IsConstraint=true    -   IsMultiValued=false    -   ConstraintType=EntityName    -   DisplayTemplateSingular=“named $1”    -   DisplayTemplatePlural=“named $1”    -   GrammaticalRole=Noun    -   Feature: RestaurantCuisine (example: “Chinese”)    -   IsConstraint=true    -   IsMultiValued=false        -   ConstraintType=CategoryConstraint    -   GrammaticalRole=AdjectiveBeforeNoun    -   Feature: RestaurantSubtype (example: “café”)    -   IsConstraint=true    -   IsMultiValued=false    -   ConstraintType=CategoryConstraint    -   DisplayTemplateSingular=“$1”    -   DisplayTemplatePlural=“$Is”    -   GrammaticalRole=Noun    -   Feature: RestaurantQualifiers (example: “romantic”)    -   IsConstraint=true    -   IsMultiValued=true    -   ConstraintType=SearchQualifier    -   DisplayTemplateSingular=“is described as $1”    -   DisplayTemplatePlural=“are described as $1”    -   DisplayTemplateCompact=“matching $1”    -   GrammaticalRole=Noun    -   Feature: FoodType (example: “burritos”)    -   IsConstraint=true    -   IsMultiValued=false    -   ConstraintType=SearchQualifier    -   DisplayTemplateSingular=“serves $1”    -   DisplayTemplatePlural=“serve $1′”    -   DisplayTemplateCompact=“serving $1”    -   GrammaticalRole=ThatClauseModifer    -   Feature: IsRecommended (example: true)    -   IsConstraint=true    -   IsMultiValued=false    -   ConstraintType=BinaryConstraint    -   DisplayTemplateSingular=“recommended”    -   DisplayTemplatePlural=“recommended”    -   GrammaticalRole=AdjectiveBeforeNoun    -   Feature: RestaurantGuessedQualifiers (example: “spectacular”)    -   IsConstraint=true    -   IsMultiValued=false    -   ConstraintType=GuessedQualifier    -   DisplayTemplateSingular=“matches $1 in reviews”    -   DisplayTemplatePlural=“match $1 in reviews”    -   DisplayTemplateCompact=“matching $1”    -   GrammaticalRole=ThatClauseModifer

In one embodiment, assistant 1002 is able to handle unmatched input. Tohandle such input, domain model 1056 can provide for nodes of typeGuessedQualifier for each selection class, and rules that matchotherwise unmatched words if they are in the right grammatical context.That is, GuessedQualifiers are treated as miscellaneous nodes in theparse which match when there are words that are not found in theontology but which are in the right context to indicate that that areprobably qualifiers of the selection class. The difference betweenGuessedQualifiers and SearchQualifiers is that the latter are matched tovocabulary in the ontology. This distinction allows us to paraphrasethat assistant 1002 identified the intent solidly on theSearchQualifiers and can be more hesitant when echoing back theGuessedQualifiers.

In one embodiment, assistant 1002 performs the following steps whengenerating paraphrase text:

-   -   1. If the task is unknown, explain what assistant 1002 can do        and prompt for more input.    -   2. If the task is a constrained selection task and the location        is known, then explain the domains that assistant 1002 knows and        prompt for the selection class.    -   3. If the selection class is known but a required constraint is        missing, then prompt for that constraint. (for example, location        is required for constrained selection on restaurants)    -   4. If the input contains an EntityName of the selection class,        then output “looking up”<name> in <location>.    -   5. If this is the initial request in a conversation, then output        “looking for” followed by the complex noun phrase that describes        the constraints.    -   6. If this is a follow-up refinement step in the dialog,        -   a. If the user just completed a required input, then output            “thanks” and then paraphrase normally. (This happens when            there is a required constraint that is mapped to the user            input.)        -   b. If the user is changing a constraint, acknowledge this            and then paraphrase normally.        -   c. If the user typed in the proper name of an instance of            the selection class, handle this specially.        -   d. If the user just added an unrecognized phrase, then            indicate how it will be folded in as search. If appropriate,            the input may be dispatched to a search service.        -   e. If the user is just adding a normal constraint, then            output “OK”, and paraphrase normally.    -   7. To explain results, use the same approach for paraphrase.        However, when the results are surprising or unexpected, then        explain the results using knowledge about the data and service.        Also, when the query is over- or underconstrained, prompt for        more input.

Grammar for Constructing Complex Noun Phrases

In one embodiment, when paraphrasing 734 a constrained selection taskquery, the foundation is a complex noun phrase around the selectionclass that refers to the current constraints. Each constraint has agrammatical position, based on its type. For example, in one embodiment,assistant 1002 may construct a paraphrase such as:

recommended romantic Italian restaurants near Menlo Park with opentables for 2 that serve osso buco and are described as “quiet” A grammarto construct this is <paraphraseNounClause> :== <binaryConstraint><searchQualifier> <categoryConstraint> <itemNoun> <locationConstraint><availabiltyConstraint> <adjectivalClauses> <binaryConstraint> :==single adjective that indicates the presence or absence of aBinaryConstraint (e.g., recommended (best), affordable (cheap)) It ispossible to list more than one in the same query. <searchQualifier> :==a word or words that match the ontology for a qualifier of the selectionclass, which would be passed into a search engine service. (e.g.,romantic restaurants, funny movies). Use when ConstraintType= Search 

 ualifier. <categoryConstraint> :== an adjective that identifies thegenre, cuisine, or category of the selection class (e.g., Chineserestaurant or R-rated file). It is the last prefix adjective because itis the most intrinsic. Use for features of type CategoryConstraint andGrammaticalRole=AdjectiveBeforeNoun. <itemNoun> :== <namedEntityPhrase>| <selectionClass> | <selectionClassSubType > find the most specific wayto display the noun. NamedEntity < SubType < Class <selectionClass > :==a noun that is the generic name for the selection class (e.g.,restaurant, movie, place) <selectionClassSubType> :== a noun phrase thatis the subtype of the selection class if it is known (e.g., diner,museum, store, bar for the selection class local business). Use forfeatures in which ConstraintType =CategoryConstraint andGrammaticalRole=AdjectiveBeforeNoun. <namedEntityPhrase> :==<entityName> | “the” (<selection Class> | <selectionClassSubType>)<entityName> :== the proper name of an instance of the selection class(e.g., “Il Fornaio” “Animal House” “Harry's Bar”) <locationConstraint>:== <locationPreposition> <locationName> <locationPreposition> :== “in”,“near”, “at”, and the like <locationName> :== city, street address,landmark, or something for OPS like “your current location”<availabilityConstraint> :== the availability constraint expressed as aprepositional phrase that comes after the noun (e.g., “with opentables”, “with seats available”, “available online”). It comes rightafter the noun to give it salience. <adjectivalClauses>:==<modiferVerbPhrase > I “that” <modiferVerbPhrase> “and”<modiferVerbPhrase> <modiferVerbPhrase> := a verb phrase that expressesa search-keyword style constraint on the selection class (e.g.,restaurants that “are described as quiet”, “serve meat after 11”, “match‘tragically hip’ in reviews”; movies that “contain violence”, “starBilly Bob Thornton”). Include all constraints in the parse whoseGrammaticalRole=ThatClauseModifer, using the “that...and” variant ifthere are more than one. Use the DisplayTemplatePlural to generate the“that” clauses, putting the GuessedQualifter last. If there is only onesuch constraint, use the DisplayTemplateCompact variant.

Table 2 provides some examples of paraphrases that may be provided inresponse to the first input for a task, according to one embodiment.

TABLE 2 Paraphrases in response to first input Input Initial Paraphrasetie house Sorry, I don't understand “tie house.” I know aboutrestaurants, hotels, and other local businesses, flights, weather, andbuying things online. What are you looking for, and where? palo alto Ican help you find restaurants, businesses, events, and things to do inPalo Alto. What are you looking for? food Looking for a place to eat?Where? (examples) Italian food I can tell you about Italian restaurantsin your area. Where would you like to eat? burrito I can help you findplaces that serve burritos in your area. Where would you like to eat?best mexican food I can help you find good Mexican restaurants in yourarea. Where are you? restaurants in palo alto Looking for restaurantsnear Palo Alto. Italian palo alto Looking for Italian restaurants nearPalo Alto. thai food in menlo park Looking for Thai restaurants nearMenlo Park Indonesian food in menlo park Looking for Indonesianrestaurants near Menlo Park best mexican food in Palo Alto Looking forrecommended Mexican restaurants near Palo Alto. fast food menlo parkLooking for fast food restaurants near Menlo Park healthy food in menlopark Looking for places near Menlo Park that serve healthy food. divebar in menlo park Looking for dive bars near Menlo Park. pool room inmenlo park Looking for pool rooms near Menlo Park. hookah room in menlopark Looking for places near Menlo Park described as “hookah room”.spicy food in sf Looking for places near Menlo Park that servespicy food. burgers in Menlo park Looking for places near Menlo Parkthat serve burgers. belly dancing place in sf Looking for places nearSan Francisco described as “belly dancing”. belly dancing place in menlopark Looking for places near Menlo Park described as “belly dancing”.juke joint in sf Looking for joints near San Francisco described as“juke”. tie house Menlo park Looking for places near Menlo Parkdescribed as “tie house”. thai house palo alto Looking up Thai House inPalo Alto ii fornaio menlo park Looking up Il Fornaio in Palo Alto iifornao in Menlo park Looking for places near Palo Alto described as “ilfornao”. cliff house in Menlo park Looking up Cliff House in Palo Altoromantic restaurant in palo alto Looking for romantic restaurants inPalo Alto romantic restaurant in palo alto Looking for romanticrestaurants in Palo Alto quiet romantic restaurant in palo alto Lookingfor romantic restaurants in Palo Alto described as “quiet” quietromantic restaurant in palo alto Looking for romantic restaurants inPalo Alto described as “quiet tragically hip restaurant in palo altoLooking for restaurants in Palo Alto described as “tragically hip”tragically hip restaurant in palo alto Looking for restaurants inPalo Alto described as “tragically hip” table for two tonight in paloalto Checking availability for tonight at restaurants near Palo Alto.table for two italian food tonight in Checking availability for tonightat Italian Palo alto restaurants near Palo Alto. il fornaio palo alto 6pm for 2 Checking availability for tables tonight at Il Fornaio inPalo Alto. table for 2 at thai house in PA Checking availability fortables tonight at Thai House in Palo Alto. Pizza delivery in menlo parkI want the best quiet romantic OK, looking for recommended romanticItalian italian restaurant for osso buco in restaurants near Menlo Parkthat serve osso buco or near menlo park and are described as “quiet”.Book me a table for 2 at the best OK, looking for recommended romanticitalian quiet romantic italian restaurant for restaurants nearMenlo Park with open tables for 2 osso buco in or near menlo park thatserve osso buco and are described as “quiet”.Refining a Query about Places to Eat

Table 3 provides some examples of paraphrases in response to a situationwhere user's intent to find a place to eat is known, but they have notselected a particular place to eat. They are facing a list ofrestaurants, and they provide refinements.

TABLE 3 Paraphrases in response to refinements Refinements menlo parkThanks, now looking for restaurants in (was un-known) Menlo Park.cheapest OK, looking for affordable restaurants near Menlo Park. ChineseOK, changing cuisine to Chinese. Looking for (was italian) Chineserestaurants near Menlo Park pizza (adding OK, looking for Italianrestaurants near food item) Menlo Park that serve pizza. san franciscoOK, changing location to San Francisco. Looking (was menlo park) forItalian restaurants near San Francisco that serve pizza. tragically hipOK, I'll search reviews for “tragically hip”. quiet romantic OK. lookingfor romantic restaurants in Palo Alto described as “quiet” cliff houseCliff House sounds familiar. I'll look it up in Palo Alto.

Table 4 provides some examples of results summaries that are providedonce results are obtained.

TABLE 4 Results Summaries Input #Results Results Summary tie house N/AN/A (no call to services) palo alto N/A N/A (no call to services) foodN/A N/A (no call to services) Italian food N/A N/A (no call to services)burrito N/A N/A (no call to services) best mexican food N/A N/A (no callto services) restaurants palo alto many Here are some recommendedrestaurants near Palo Alto. Do you prefer any particular food type?Would you like a reservation? Italian palo alto some Here are someItalian restaurants near Palo Alto. To reserve a table, click onavailable times. thai food in menlo park some Here are some Thairestaurants near Menlo Park. To reserve a table, click on availabletimes. indonesian food in menlo 0 Sorry I can't find any Indonesianrestaurants near park Menlo Park. You could try other cuisines orlocations. best mexican food in Palo some Here are some recommendedMexican restaurants Alto near Palo Alto. fast food menlo park some OK,here are some fast food restaurants near Menlo Park. healthy food inmenlo some Here are some places near Menlo Park that might park senehealthy food, based on reviews. dive bar in menlo park some Here aresome bars near Menlo Park. I don't see anything specific to “dive bars,”Are there other ways of saying it? pool room in menlo park some Here aresome places in Menlo Park that are described as pool rooms. You can addother words to describe what you want. hookah room in menlo 0 I can'tfind places Menlo Park matching “hookah park room”. Interested inrestaurants, hotels, or local businesses? spicy food in sf weak Here aresome places to eat in San Francisco. I match don't know which servespicy food. Could it go by another name? burgers in Menlo park some Youmight find burgers at these places to eat near Menlo Park. belly dancingplace in sf some Here are some places in Menlo Park that are describedas “belly dancing”. You can add other words to describe what you wantbelly dancing place in 0 I can help you find restaurants, businesses,events, menlo park and things to do in San Francisco, What are someother words for “belly dancing place”? juke joint in sf 0 I can help youfind restaurants, businesses, events, and things to do in San Francisco.What are some other words for “juke joint”? tie house Menlo park 0Sorry, I don't know about “tie house” in Menlo Park. Any other names forit? thai house palo alto 1 Got it: Thai House restaurant inPalo Alto, CA. Want to reserve a table? When and for how many? ilfornaio menlo park weak There is a Il Fornaio in Palo Alto, which isnear match Menlo Park. il fornao in Menlo park spell Did you meanIl Fornaio in Palo Alto? If not, try match different words. cliff housein Menlo park 0 I couldn't find Cliff House in Palo Alto. Could it go bydifferent name or be in a different city? romantic restaurant in someHere are some restaurants near Palo Alto that are palo alto described inreviews as romantic. romantic restaurant in 0 Here are restaurants nearPalo Alto. Searching palo alto reviews did not find any described asromantic. Any other preferences? quiet romantic restaurant some Here aresome restaurants near Palo Alto that match in palo alto “quiet romantic”in reviews. quiet romantic restaurant 0 Here are some restaurants nearPalo Alto. in palo alto Searching reviews did find any matching “quietromantic”. Any other preferences? tragically hip restaurant some Hereare some restaurants near Palo Alto that match in palo alto “tragicallyhip” in reviews. tragically hip restaurant 0 Here are some restaurantsnear Palo Alto. in palo alto Searching reviews did not find any matching“tragically hip”. Any other preferences? table for two italian food someHere are restaurants with open tables from 6pm to tonight in palo alto8pm tonight for a party of two in Palo Alto, CA, table for two italianfood some Here are restaurants with open tables from 6pm to tonight inpalo alto 8pm tonight for a party of two in Palo Alto, CA. il fornaiopalo alto 6pm 1 Here are table options at Il Fornaio restaurant in for 2Palo Alto, CA at 6pm today for a party of 2. table for 2 at thai house 1Here is information for Thai House in Palo Alto, in PA CA. You can callthem at the phone number listed. pizza delivery in menlo Here are placesthat serve pizza in menlo park. I do park not have delivery options atthis time. Click on restaurant phone number to call for delivery. I wantthe best quiet weak Here are the best matches for recommended romanticitalian match, romantic italian restaurant near Menlo Park thatrestaurant for osso buco soiled by serve osso buco and are described as“quiet”. Click in or near menlo park availability on available times toreserve a table. Book me a table for 2 at weak Here are the best matchesfor recommended the best quiet romantic match, romantic italianrestaurants near Menlo Park with italian restaurant for osso sorted byopen tables for 2 that serve osso buco and are buco in or near menloavailability described as “quiet”. Click on available times to parkreserve a table. refinements menlo park (was un- some Here are somerecommended restaurants near known) Menlo Park. Do you prefer anyparticular food type? cheapest some I found 4 affordable restaurantsnear Menlo Park. Chinese (was Italian) some I found 4 Chineserestaurants near Menlo Park. pizza (adding food item) some I found 4Italian restaurants near Menlo Park that serve pizza. san Francisco (wasmenlo some I found 4 Italian restaurants near San Francisco. park)tragically hip some I found 4 restaurants near Palo Alto that match“tragically hip” in reviews. quiet romantic some Here are somerestaurants near Palo Alto that match “quiet romantic” in reviews. cliffhouse 0 I couldn't find Cliff House in Palo Alto. Could it go by adifferent name or be in a different city?

Table 5 provides some examples of prompts that are provided when usersclick on active links.

Prompts when Users Click on Active Links

TABLE 5 Prompts when users click on active links Anchor Text Prompt TextNotes Location, where Enter a location: a city, zip code, or This promptmight be used when “near” followed by the name of a the user has notspecified a place. location yet Palo Alto Enter a location such as acity, This prompt might be used when neighborhood, street address, orthe user is changing locations. “near” followed by a landmark. food typeEnter a food category such as Merge food type and cuisine can Chinese orPizza. be merged Italian You can try other food categories User alreadysaid Italian. such as Chinese, or a favorite food Assistant 1002 ishelping the item such as stake. user explore alter-natives. If it is afood item, if dominates over cuisine. reservation Enter the day and timeto reserve a Prompting for a reservation table, such as “tomorrow at 8”.healthy food You can also enter menu items or Known food type cuisinesspicy food You can also enter menu items or Unknown food type cuisinesrestaurants What kind of restaurant? (e.g., Clicking on the restaurantslink Chinese, Pizza) should insert the wood “restaurant” on the end ofthe text input. businesses You can find local florists, ATMs, Clickingon the businesses link doctors, drug stores, and the like should add tothe machine What kind of business are you readable tag that this is alocal looking for? search events You can discover upcoming converts,shows and the like What interests you? things to do Music, art, theater,sports, and the like What kind of thing would you like to do in thisarea? hotels I can help you find an available hotel room. Anypreferences for amenities or location? weather Enter a city, and I'lltell you what If location is known, just show the weather is like there.the weather data buying things I can help you find music, movies, books,electronics, toys and more - and buy it from Amazon. What are youlooking for?

Suggesting Possible Responses in a Dialog

In one embodiment, assistant 1002 provides contextual suggestions.Suggestions a way for assistant 1002 to offer the user options to moveforward from his or her current situation in the dialog. The set ofsuggestions offered by assistant 1002 depends on context, and the numberof suggestions offered may depend on the medium and form factor. Forexample, in one embodiment, the most salient suggestions may be offeredin line in the dialog, an extended list of suggestions (“more”) may beoffered in a scrollable menu, and even more suggestions are reachable bytyping a few characters and picking from autocomplete options. Oneskilled in the art will recognize that other mechanisms may be used forproviding suggestions.

In various embodiments, different types of suggestions may be provided.Examples of suggestion types include:

-   -   options to refine a query, including adding or removing or        changing constraint values;    -   options to repair or recover from bad situations, such as “not        what I mean” or “start over” or “search the web”;    -   options to disambiguate among;    -   interpretations of speech;    -   interpretations of text, including spell correction and semantic        ambiGUIty;    -   context-specific commands, such as “show these on a map” or        “send directions to my date” or “explain these results”;    -   suggested cross-selling offers, such as next steps in meal or        event planning scenarios;    -   options to reuse previous commands, or parts of them.

In various embodiments, the context that determines the most relevantsuggestions may be derived from, for example:

-   -   dialog state    -   user state, including, for example:        -   static properties (name, home address, etc)        -   dynamic properties (location, time, network speed)    -   interaction history, including, for example:        -   query history        -   results history        -   the text that has been entered so far into autocomplete.

In various embodiments, suggestions may be generated by any mechanism,such as for example:

-   -   paraphrasing a domain, task, or constraint based on the ontology        model;    -   prompting in autocomplete based on the current domain and        constraints;    -   paraphrasing ambiguous alternative interpretations;    -   alternative interpretations of speech-to-text;    -   hand authoring, based on special dialog conditions.

According to one embodiment, suggestions are generated as operations oncommands in some state of completion. Commands are explicit, canonicalrepresentations of requests, including assumptions and inferences, basedon attempted interpretations on user input. In situations where the userinput is incomplete or ambiguous, suggestions are an attempt to help theuser adjust the input to clarify the command.

In one embodiment, each command is an imperative sentence having somecombination of a

-   -   command verb (imperative such as “find” or “where is”);    -   domain (selection class such as “restaurants”);    -   constraint(s) such as location=Palo Alto and cuisine=Italian.

These parts of a command (verb, domain, constraints) correspond to nodesin the ontology.

A suggestion, then, may be thought of as operations on a command, suchas setting it, changing it, or declaring that it is relevant or notrelevant. Examples include:

-   -   setting a command verb or domain (“find restaurants”)    -   changing a command verb (“book it”, “map it”, “save it”)    -   changing a domain (“looking for a restaurant, not a local        business”)    -   stating that a constraint is relevant (“try refining by        cuisine”)    -   choosing a value for a constraint (“Italian”, “French”, and the        like)    -   choosing a constraint and value together (“near here”, “tables        for 2”)    -   stating that a constraint value is wrong (“not that Boston”)    -   stating that a constraint is not relevant (“ignore the expense”)    -   stating the intent to change a constraint value (“try a        different location”)    -   changing a constraint value (“Italian, not Chinese”)    -   adding to a constraint value (“and with a pool, too”)    -   snapping a value to grid (“Los Angeles, not los angelos”)    -   initiating a new command, reusing context ([after movies] “find        nearby restaurants”, “send directions to my friend”)    -   initiating a command that is “meta” to context (“explain these        results”)    -   initiating a new command, resetting or ignoring context (“start        over”, “help with speech”)

A suggestion may also involve some combination of the above. Forexample:

-   -   “the movie Milk not [restaurants serving] the food item milk”    -   “restaurants serving pizza, not just pizza joints”    -   “The place called Costco in Mountain View, I don't care whether        you think it is a restaurant or local business”    -   “Chinese in mountain view” [a recent query]

In one embodiment, assistant 1002 includes a general mechanism tomaintain a list of suggestions, ordered by relevance. The format inwhich a suggestion is offered may differ depending on current context,mode, and form factor of the device.

In one embodiment, assistant 1002 determines which constraints to modifyby considering any or all of the following factors:

-   -   Consider whether the constraint has a value;    -   Consider whether the constraint was inferred or explicitly        stated;    -   Consider its salience (suggestionIndex).

In one embodiment, assistant 1002 determines an output format for thesuggestion. Examples of output formats include:

-   -   change domain:        -   if autocomplete option “find restaurants”, then “try            something different”        -   else [was inferred] “not looking for restaurants”    -   change name constraint:        -   if name was inferred, offer alterative ambiguous            interpretation”        -   stuff into autocomplete the entity names from current            results        -   different name        -   consider that it wasn't a name lookup (remove            constraint)—maybe offer category in place of it    -   “not named”    -   “not in Berkeley”    -   “some other day”    -   not that sense of (use ambiguity alternatives)    -   inferred date: “any day, I don't need a reservation”

In one embodiment, assistant 1002 attempts to resolve ambiguities viasuggestions. For example, if the set of current interpretations of userintent is too ambiguous 310, then suggestions are one way to prompt formore information 322. In one embodiment, for constrained selectiontasks, assistant 1002 factors out common constraints among ambiguousinterpretations of intent 290 and presents the differences among them tothe user. For example, if the user input includes the word “café” andthis word could match the name of a restaurant or the type ofrestaurant, then assistant 102 can ask “did you mean restaurants named‘café’ or ‘café restaurants’?”

In one embodiment, assistant 1002 infers constraints under certainsituations. That is, for constrained selection tasks, not allconstraints need be mentioned explicitly in the user input; some can beinferred from other information available in active ontology 1050, shortterm memory 1052, and/or other sources of information available toassistant 1002. For example:

-   -   Inferring domain or location    -   Default assumption, like location    -   Weakly matched constraint (fuzzy, low salience location, etc)    -   Ambiguous criteria (match to constraint value without prefix        (name vs. category, often ambiguous)

In cases where the assistant 1002 infers constraint values, it may alsooffer these assumptions as suggestions for the user to overrule. Forexample, it might tell the user “I assumed you meant around here. Wouldyou like to look at a different location?”

The present invention has been described in particular detail withrespect to possible embodiments. Those of skill in the art willappreciate that the invention may be practiced in other embodiments.First, the particular naming of the components, capitalization of terms,the attributes, data structures, or any other programming or structuralaspect is not mandatory or significant, and the mechanisms thatimplement the invention or its features may have different names,formats, or protocols. Further, the system may be implemented via acombination of hardware and software, as described, or entirely inhardware elements, or entirely in software elements. Also, theparticular division of functionality between the various systemcomponents described herein is merely exemplary, and not mandatory;functions performed by a single system component may instead beperformed by multiple components, and functions performed by multiplecomponents may instead be performed by a single component.

In various embodiments, the present invention can be implemented as asystem or a method for performing the above-described techniques, eithersingly or in any combination. In another embodiment, the presentinvention can be implemented as a computer program product comprising anontransitory computer-readable storage medium and computer programcode, encoded on the medium, for causing a processor in a computingdevice or other electronic device to perform the above-describedtechniques.

Reference in the specification to “one embodiment” or to “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiments is included in at least oneembodiment of the invention. The appearances of the phrase “in oneembodiment” in various places in the specification are not necessarilyall referring to the same embodiment.

Some portions of the above are presented in terms of algorithms andsymbolic representations of operations on data bits within a memory of acomputing device. These algorithmic descriptions and representations arethe means used by those skilled in the data processing arts to mosteffectively convey the substance of their work to others skilled in theart. An algorithm is here, and generally, conceived to be aself-consistent sequence of steps (instructions) leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical, magnetic or optical signals capable of being stored,transferred, combined, compared and otherwise manipulated. It isconvenient at times, principally for reasons of common usage, to referto these signals as bits, values, elements, symbols, characters, terms,numbers, or the like. Furthermore, it is also convenient at times, torefer to certain arrangements of steps requiring physical manipulationsof physical quantities as modules or code devices, without loss ofgenerality.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“displaying” or “determining” or the like, refer to the action andprocesses of a computer system, or similar electronic computing moduleand/or device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system memories orregisters or other such information storage, transmission or displaydevices.

Certain aspects of the present invention include process steps andinstructions described herein in the form of an algorithm. It should benoted that the process steps and instructions of the present inventioncan be embodied in software, firmware and/or hardware, and when embodiedin software, can be downloaded to reside on and be operated fromdifferent platforms used by a variety of operating systems.

The present invention also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general-purpose computing deviceselectively activated or reconfigured by a computer program stored inthe computing device. Such a computer program may be stored in acomputer readable storage medium, such as, but is not limited to, anytype of disk including floppy disks, optical disks, CD-ROMs,magnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, applicationspecific integrated circuits (ASICs), or any type of media suitable forstoring electronic instructions, and each coupled to a computer systembus. Further, the computing devices referred to herein may include asingle processor or may be architectures employing multiple processordesigns for increased computing capability.

The algorithms and displays presented herein are not inherently relatedto any particular computing device, virtualized system, or otherapparatus. Various general-purpose systems may also be used withprograms in accordance with the teachings herein, or it may proveconvenient to construct more specialized apparatus to perform therequired method steps. The required structure for a variety of thesesystems will be apparent from the description provided herein. Inaddition, the present invention is not described with reference to anyparticular programming language. It will be appreciated that a varietyof programming languages may be used to implement the teachings of thepresent invention as described herein, and any references above tospecific languages are provided for disclosure of enablement and bestmode of the present invention.

Accordingly, in various embodiments, the present invention can beimplemented as software, hardware, and/or other elements for controllinga computer system, computing device, or other electronic device, or anycombination or plurality thereof. Such an electronic device can include,for example, a processor, an input device (such as a keyboard, mouse,touchpad, trackpad, joy-stick, trackball, microphone, and/or anycombination thereof), an output device (such as a screen, speaker,and/or the like), memory, long-term storage (such as magnetic storage,optical storage, and/or the like), and/or network connectivity,according to techniques that are well known in the art. Such anelectronic device may be portable or nonportable. Examples of electronicdevices that may be used for implementing the invention include: amobile phone, personal digital assistant, smartphone, kiosk, desktopcomputer, laptop computer, tablet computer, consumer electronic device,consumer entertainment device; music player; camera; television; set-topbox; electronic gaming unit; or the like. An electronic device forimplementing the present invention may use any operating system such as,for example, iOS or MacOS, available from Apple Inc. of Cupertino,Calif., or any other operating system that is adapted for use on thedevice.

While the invention has been described with respect to a limited numberof embodiments, those skilled in the art, having benefit of the abovedescription, will appreciate that other embodiments may be devised whichdo not depart from the scope of the present invention as describedherein. In addition, it should be noted that the language used in thespecification has been principally selected for readability andinstructional purposes, and may not have been selected to delineate orcircumscribe the inventive subject matter. Accordingly, the disclosureof the present invention is intended to be illustrative, but notlimiting, of the scope of the invention, which is set forth in theclaims.

What is claimed is:
 1. An electronic device, comprising: one or moreprocessors; a memory; and one or more programs, wherein the one or moreprograms are stored in the memory and configured to be executed by theone or more processors, the one or more programs including instructionsfor: receiving input from a user; determining an intent corresponding tothe received input; performing a first task based on the determinedintent; providing a first output associated with the performed task; inaccordance with a detection of a user selection corresponding to thefirst output: providing a second output including at least oneadditional task corresponding to the performed task; and in accordancewith a detection that no user selection corresponding to the firstoutput has occurred: providing a third output including at least oneoption to select a task parameter associated with the first task.
 2. Thedevice of claim 1, the one or more programs further includinginstructions for: receiving a second input from the user, wherein thesecond input corresponds to a selection of a task from the at least oneprovided additional task; and performing a second task based on thesecond input.
 3. The device of claim 1, wherein the at least oneadditional task includes an interaction with a remote source over anetwork connection.
 4. The device of claim 1, wherein the at least oneadditional task includes an interaction with information stored on theelectronic device.
 5. The device of claim 1, wherein the at least oneadditional task includes a task to save information corresponding to thefirst task.
 6. The device of claim 1, wherein the at least oneadditional task includes a task to send information to another user,wherein the information corresponds to the first task.
 7. The device ofclaim 1, wherein the at least one additional task includes a task todisplay map information corresponding to the first task.
 8. The deviceof claim 1, wherein the at least one additional task includes a task toinitiate a phone call to a contact associated with the first task. 9.The device of claim 1, wherein the at least one additional task includesa task to set an alarm associated with the first task.
 10. The device ofclaim 1, wherein the at least one additional task includes a task toinitiate a reservation procedure associated with the first task.
 11. Thedevice of claim 1, wherein the at least one additional task includes atask to create a note file associated with the first task.
 12. Thedevice of claim 1, wherein the second output includes an audiblerepresentation of at least one additional task corresponding to theperformed task.
 13. The device of claim 1, the one or more programsfurther including instructions for: receiving a second user inputincluding a speech input; and determining a user intent of the secondspeech input based on the second output including at least oneadditional task corresponding to the performed task.
 14. The device ofclaim 1, wherein at least one of the first task and the at least oneadditional task includes performing a task on a secondary device. 15.The device of claim 1, wherein the task parameter is associated with adomain corresponding to the first task.
 16. The device of claim 1,wherein determining an intent corresponding to the received inputcomprises: obtaining context information including a history ofinteractions between the user and the electronic device; and determiningthe intent based on the context information.
 17. The device of claim 16,the one or more programs further including instructions for: prior toreceiving the input from the user, receiving a speech input from theuser within a predetermined time of the received input; and processingthe received speech input to obtain the context information.
 18. Thedevice of claim 16, the one or more programs further includinginstructions for: prior to receiving the input from the user, receivinga non-speech input from the user within a predetermined time of thereceived input; and processing the received non-speech input to obtainthe context information.
 19. A non-transitory computer readable storagemedium storing one or more programs, the one or more programs comprisinginstructions, which when executed by one or more processors of anelectronic device, cause the device to: receive input from a user;determine an intent corresponding to the received input; perform a firsttask based on the determined intent; provide a first output associatedwith the performed task; in accordance with a detection of a userselection corresponding to the first output: provide a second outputincluding at least one additional task corresponding to the performedtask; and in accordance with a detection that no user selectioncorresponding to the first output has occurred: provide a third outputincluding at least one option to select a task parameter associated withthe first task.
 20. The computer readable storage medium of claim 19,wherein the one or more programs cause the device to: receive a secondinput from the user, wherein the second input corresponds to a selectionof a task from the at least one provided additional task; and perform asecond task based on the second input.
 21. The computer readable storagemedium of claim 19, wherein the at least one additional task includes aninteraction with a remote source over a network connection.
 22. Thecomputer readable storage medium of claim 19, wherein the at least oneadditional task includes an interaction with information stored on theelectronic device.
 23. The computer readable storage medium of claim 19,wherein the at least one additional task includes a task to saveinformation corresponding to the first task.
 24. The computer readablestorage medium of claim 19, wherein the at least one additional taskincludes a task to send information to another user, wherein theinformation corresponds to the first task.
 25. The computer readablestorage medium of claim 19, wherein the at least one additional taskincludes a task to display map information corresponding to the firsttask.
 26. The computer readable storage medium of claim 19, wherein theat least one additional task includes a task to initiate a phone call toa contact associated with the first task.
 27. The computer readablestorage medium of claim 19, wherein the at least one additional taskincludes a task to set an alarm associated with the first task.
 28. Thecomputer readable storage medium of claim 19, wherein the at least oneadditional task includes a task to initiate a reservation procedureassociated with the first task.
 29. The computer readable storage mediumof claim 19, wherein the at least one additional task includes a task tocreate a note file associated with the first task.
 30. The computerreadable storage medium of claim 19, wherein the second output includesan audible representation of at least one additional task correspondingto the performed task.
 31. The computer readable storage medium of claim19, wherein the one or more programs cause the device to: receive asecond user input including a speech input; and determine a user intentof the second speech input based on the second output including at leastone additional task corresponding to the performed task.
 32. Thecomputer readable storage medium of claim 19, wherein at least one ofthe first task and the at least one additional task includes performinga task on a secondary device.
 33. The computer readable storage mediumof claim 19, wherein the task parameter is associated with a domaincorresponding to the first task.
 34. The computer readable storagemedium of claim 19, wherein determining an intent corresponding to thereceived input comprises: obtaining context information including ahistory of interactions between the user and the electronic device; anddetermining the intent based on the context information.
 35. Thecomputer readable storage medium of claim 34, wherein the one or moreprograms cause the device to: prior to receiving the input from theuser, receiving a speech input from the user within a predetermined timeof the received input; and processing the received speech input toobtain the context information.
 36. The computer readable storage mediumof claim 34, wherein the one or more programs cause the device to: priorto receiving the input from the user, receiving a non-speech input fromthe user within a predetermined time of the received input; andprocessing the received non-speech input to obtain the contextinformation.
 37. A computer-implemented method, comprising: at anelectronic device with one or more processors and memory: receivinginput from a user; determining an intent corresponding to the receivedinput; performing a first task based on the determined intent; providinga first output associated with the performed task; in accordance with adetection of a user selection corresponding to the first output:providing a second output including at least one additional taskcorresponding to the performed task; and in accordance with a detectionthat no user selection corresponding to the first output has occurred:providing a third output including at least one option to select a taskparameter associated with the first task.
 38. The method of claim 37,comprising: receiving a second input from the user, wherein the secondinput corresponds to a selection of a task from the at least oneprovided additional task; and performing a second task based on thesecond input.
 39. The method of claim 37, wherein the at least oneadditional task includes an interaction with a remote source over anetwork connection.
 40. The method of claim 37, wherein the at least oneadditional task includes an interaction with information stored on theelectronic device.
 41. The method of claim 37, wherein the at least oneadditional task includes a task to save information corresponding to thefirst task.
 42. The method of claim 37, wherein the at least oneadditional task includes a task to send information to another user,wherein the information corresponds to the first task.
 43. The method ofclaim 37, wherein the at least one additional task includes a task todisplay map information corresponding to the first task.
 44. The methodof claim 37, wherein the at least one additional task includes a task toinitiate a phone call to a contact associated with the first task. 45.The method of claim 37, wherein the at least one additional taskincludes a task to set an alarm associated with the first task.
 46. Themethod of claim 37, wherein the at least one additional task includes atask to initiate a reservation procedure associated with the first task.47. The method of claim 37, wherein the at least one additional taskincludes a task to create a note file associated with the first task.48. The method of claim 37, wherein the second output includes anaudible representation of at least one additional task corresponding tothe performed task.
 49. The method of claim 37, comprising: receiving asecond user input including a speech input; and determining a userintent of the second speech input based on the second output includingat least one additional task corresponding to the performed task. 50.The method of claim 37, wherein at least one of the first task and theat least one additional task includes performing a task on a secondarydevice.
 51. The method of claim 37, wherein the task parameter isassociated with a domain corresponding to the first task.
 52. The methodof claim 37, wherein determining an intent corresponding to the receivedinput comprises: obtaining context information including a history ofinteractions between the user and the electronic device; and determiningthe intent based on the context information.
 53. The method of claim 52,comprising: prior to receiving the input from the user, receiving aspeech input from the user within a predetermined time of the receivedinput; and processing the received speech input to obtain the contextinformation.
 54. The method of claim 52, comprising: prior to receivingthe input from the user, receiving a non-speech input from the userwithin a predetermined time of the received input; and processing thereceived non-speech input to obtain the context information.